Login
[x]
Log in using an account from:
Fedora Account System
Red Hat Associate
Red Hat Customer
Or login using a Red Hat Bugzilla account
Forgot Password
Login:
Hide Forgot
Create an Account
Red Hat Bugzilla – Attachment 944983 Details for
Bug 1150525
When ISO domain is attached to two DCs it appears twice in the storage tab
[?]
New
Simple Search
Advanced Search
My Links
Browse
Requests
Reports
Current State
Search
Tabular reports
Graphical reports
Duplicates
Other Reports
User Changes
Plotly Reports
Bug Status
Bug Severity
Non-Defaults
|
Product Dashboard
Help
Page Help!
Bug Writing Guidelines
What's new
Browser Support Policy
5.0.4.rh83 Release notes
FAQ
Guides index
User guide
Web Services
Contact
Legal
This site requires JavaScript to be enabled to function correctly, please enable it.
engine log
engine.log (text/plain), 16.58 MB, created by
Petr Beňas
on 2014-10-08 12:50:52 UTC
(
hide
)
Description:
engine log
Filename:
MIME Type:
Creator:
Petr Beňas
Created:
2014-10-08 12:50:52 UTC
Size:
16.58 MB
patch
obsolete
> at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:34:44,397 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:34:44,397 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-39) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:34:44,399 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-39) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 08:34:44,399 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-39) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:34:44,406 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-39) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:34:47,414 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:34:48,399 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:34:48,399 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:34:48,399 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-78) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:34:48,401 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-78) [4e91cc97] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 08:34:48,401 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-78) [4e91cc97] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:34:48,407 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-78) [4e91cc97] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:34:51,414 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:34:53,414 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:34:53,414 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:34:53,414 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-40) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:34:53,415 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-40) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 08:34:53,415 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-40) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:34:53,423 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-40) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:34:56,430 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:34:57,417 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:34:57,417 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:34:57,417 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-22) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:34:57,419 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-22) [1e1b6a16] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 08:34:57,419 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-22) [1e1b6a16] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:34:57,428 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-22) [1e1b6a16] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:35:00,435 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:35:02,435 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:35:02,435 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:35:02,435 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-46) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:35:02,436 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-46) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 08:35:02,436 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-46) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:35:02,444 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-46) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:35:05,452 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:35:06,439 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:35:06,439 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:35:06,439 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-99) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:35:06,440 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-99) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 08:35:06,440 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-99) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:35:06,449 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-99) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:35:09,456 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:35:11,457 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:35:11,457 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:35:11,457 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-64) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:35:11,458 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-64) [785bb27f] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 08:35:11,458 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-64) [785bb27f] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:35:11,466 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-64) [785bb27f] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:35:14,480 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:35:15,459 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:35:15,459 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:35:15,459 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-31) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:35:15,461 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-31) [2eda49c0] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 08:35:15,461 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-31) [2eda49c0] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:35:15,471 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-31) [2eda49c0] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:35:18,479 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:35:20,479 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:35:20,479 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:35:20,479 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-79) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:35:20,481 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-79) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 08:35:20,481 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-79) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:35:20,491 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-79) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:35:23,499 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:35:24,481 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:35:24,481 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:35:24,481 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-81) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:35:24,483 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-81) [2dc3b30d] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 08:35:24,483 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-81) [2dc3b30d] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:35:24,491 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-81) [2dc3b30d] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:35:27,502 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:35:29,502 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:35:29,502 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:35:29,502 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-19) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:35:29,504 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-19) [5a26db10] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 08:35:29,504 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-19) [5a26db10] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:35:29,514 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-19) [5a26db10] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:35:32,523 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:35:33,504 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:35:33,504 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:35:33,504 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-80) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:35:33,505 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-80) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 08:35:33,505 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-80) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:35:33,520 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-80) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:35:36,526 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:35:38,526 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:35:38,526 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:35:38,526 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-62) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:35:38,527 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-62) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 08:35:38,527 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-62) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:35:38,536 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-62) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:35:41,544 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:35:42,528 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:35:42,528 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:35:42,528 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-7) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:35:42,529 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-7) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 08:35:42,529 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-7) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:35:42,537 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-7) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:35:45,545 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:35:47,546 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:35:47,546 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:35:47,546 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-77) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:35:47,547 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-77) [22f3702] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 08:35:47,547 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-77) [22f3702] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:35:47,556 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-77) [22f3702] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:35:50,564 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:35:51,548 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:35:51,548 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:35:51,548 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-76) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:35:51,550 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-76) [1bdd13e4] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 08:35:51,550 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-76) [1bdd13e4] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:35:51,561 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-76) [1bdd13e4] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:35:54,571 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:35:56,571 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:35:56,571 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:35:56,571 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-36) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:35:56,573 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-36) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 08:35:56,573 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-36) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:35:56,580 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-36) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:35:59,588 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:36:00,573 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:36:00,573 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:36:00,573 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-33) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:36:00,575 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-33) [451c3558] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 08:36:00,575 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-33) [451c3558] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:36:00,583 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-33) [451c3558] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:36:03,589 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:36:05,589 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:36:05,589 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:36:05,589 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-35) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:36:05,591 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-35) [1ab5e2c1] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 08:36:05,591 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-35) [1ab5e2c1] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:36:05,601 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-35) [1ab5e2c1] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:36:08,609 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:36:09,591 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:36:09,591 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:36:09,591 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-57) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:36:09,593 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-57) [16b48795] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 08:36:09,593 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-57) [16b48795] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:36:09,601 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-57) [16b48795] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:36:12,610 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:36:14,610 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:36:14,610 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:36:14,612 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-56) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:36:14,613 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-56) [60b15432] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 08:36:14,613 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-56) [60b15432] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:36:14,620 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-56) [60b15432] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:36:17,628 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:36:18,612 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:36:18,612 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:36:18,613 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-14) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:36:18,614 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-14) [6343caea] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 08:36:18,614 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-14) [6343caea] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:36:18,623 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-14) [6343caea] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:36:21,631 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:36:23,631 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:36:23,631 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:36:23,631 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-42) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:36:23,633 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-42) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 08:36:23,633 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-42) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:36:23,640 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-42) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:36:26,646 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:36:27,633 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:36:27,633 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:36:27,633 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-17) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:36:27,635 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-17) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 08:36:27,635 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-17) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:36:27,642 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-17) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:36:30,648 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:36:32,648 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:36:32,648 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:36:32,648 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-20) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:36:32,649 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-20) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 08:36:32,649 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-20) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:36:32,656 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-20) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:36:35,663 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:36:36,650 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:36:36,650 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:36:36,650 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-74) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:36:36,651 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-74) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 08:36:36,651 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-74) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:36:36,660 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-74) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:36:39,669 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:36:41,669 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:36:41,669 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:36:41,669 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-36) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:36:41,671 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-36) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 08:36:41,671 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-36) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:36:41,678 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-36) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:36:44,685 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:36:45,671 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:36:45,671 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:36:45,671 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-87) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:36:45,673 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-87) [1dbf513e] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 08:36:45,673 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-87) [1dbf513e] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:36:45,693 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-87) [1dbf513e] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:36:48,700 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:36:50,701 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:36:50,701 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:36:50,701 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-35) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:36:50,703 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-35) [1ab5e2c1] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 08:36:50,703 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-35) [1ab5e2c1] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:36:50,709 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-35) [1ab5e2c1] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:36:53,716 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:36:54,703 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:36:54,703 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:36:54,703 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-37) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:36:54,705 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-37) [310dcfa0] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 08:36:54,705 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-37) [310dcfa0] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:36:54,712 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-37) [310dcfa0] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:36:57,720 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:36:59,720 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:36:59,720 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:36:59,720 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-56) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:36:59,721 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-56) [60b15432] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 08:36:59,722 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-56) [60b15432] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:36:59,730 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-56) [60b15432] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:37:02,738 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:37:03,722 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:37:03,722 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:37:03,722 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-14) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:37:03,724 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-14) [6343caea] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 08:37:03,724 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-14) [6343caea] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:37:03,732 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-14) [6343caea] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:37:03,738 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-14) [6343caea] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:37:06,746 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:37:08,746 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:37:08,746 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:37:08,746 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-94) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:37:08,747 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-94) [5dd0314d] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 08:37:08,747 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-94) [5dd0314d] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:37:08,755 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-94) [5dd0314d] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:37:11,762 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:37:12,750 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:37:12,750 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:37:12,750 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-97) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:37:12,752 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-97) [54081188] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 08:37:12,752 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-97) [54081188] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:37:12,759 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-97) [54081188] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:37:15,764 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:37:17,765 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:37:17,765 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:37:17,765 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-11) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:37:17,767 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-11) [4df5b8de] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 08:37:17,767 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-11) [4df5b8de] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:37:17,775 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-11) [4df5b8de] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:37:20,783 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:37:21,767 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:37:21,767 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:37:21,767 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-74) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:37:21,769 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-74) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 08:37:21,769 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-74) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:37:21,777 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-74) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:37:24,797 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:37:26,797 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:37:26,797 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:37:26,797 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-33) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:37:26,798 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-33) [451c3558] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 08:37:26,799 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-33) [451c3558] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:37:26,807 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-33) [451c3558] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:37:29,814 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:37:30,805 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:37:30,805 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:37:30,805 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-16) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:37:30,806 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-16) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 08:37:30,806 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-16) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:37:30,813 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-16) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:37:33,821 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:37:35,821 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:37:35,821 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:37:35,821 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-39) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:37:35,822 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-39) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 08:37:35,822 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-39) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:37:35,831 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-39) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:37:38,839 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:37:39,823 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:37:39,823 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:37:39,823 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-12) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:37:39,826 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-12) [7458e5a2] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 08:37:39,826 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-12) [7458e5a2] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:37:39,844 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-12) [7458e5a2] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:37:42,853 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:37:44,853 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:37:44,853 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:37:44,854 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-67) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:37:44,856 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-67) [6cea7f79] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 08:37:44,856 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-67) [6cea7f79] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:37:44,863 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-67) [6cea7f79] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:37:47,870 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:37:48,855 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:37:48,855 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:37:48,855 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-42) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:37:48,857 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-42) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 08:37:48,857 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-42) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:37:48,865 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-42) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:37:51,874 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:37:53,874 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:37:53,874 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:37:53,874 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-46) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:37:53,875 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-46) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 08:37:53,875 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-46) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:37:53,883 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-46) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:37:56,891 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:37:57,876 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:37:57,876 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:37:57,876 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-30) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:37:57,877 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-30) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 08:37:57,877 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-30) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:37:57,886 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-30) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:38:00,892 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:38:02,892 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:38:02,892 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:38:02,892 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-76) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:38:02,893 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-76) [1bdd13e4] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 08:38:02,893 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-76) [1bdd13e4] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:38:02,901 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-76) [1bdd13e4] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:38:05,908 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:38:06,895 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:38:06,898 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:38:06,898 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-31) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:38:06,900 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-31) [2eda49c0] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 08:38:06,900 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-31) [2eda49c0] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:38:06,907 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-31) [2eda49c0] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:38:09,913 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:38:11,914 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:38:11,914 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:38:11,915 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-33) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:38:11,917 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-33) [451c3558] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 08:38:11,917 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-33) [451c3558] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:38:11,924 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-33) [451c3558] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:38:14,932 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:38:15,916 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:38:15,916 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:38:15,916 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-3) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:38:15,918 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-3) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 08:38:15,918 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-3) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:38:15,925 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-3) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:38:18,932 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:38:20,932 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:38:20,932 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:38:20,932 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-37) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:38:20,934 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-37) [310dcfa0] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 08:38:20,934 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-37) [310dcfa0] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:38:20,941 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-37) [310dcfa0] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:38:23,950 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:38:24,934 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:38:24,934 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:38:24,934 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-84) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:38:24,935 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-84) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 08:38:24,935 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-84) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:38:24,944 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-84) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:38:27,951 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:38:29,951 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:38:29,951 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:38:29,953 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-15) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:38:29,954 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-15) [3a4dd3fc] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 08:38:29,955 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-15) [3a4dd3fc] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:38:29,963 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-15) [3a4dd3fc] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:38:32,972 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:38:33,953 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:38:33,953 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:38:33,953 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-38) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:38:33,955 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-38) [56c37b5] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 08:38:33,955 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-38) [56c37b5] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:38:33,967 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-38) [56c37b5] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:38:36,975 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:38:38,975 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:38:38,975 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:38:38,975 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-77) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:38:38,976 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-77) [22f3702] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 08:38:38,976 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-77) [22f3702] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:38:38,984 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-77) [22f3702] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:38:41,991 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:38:42,978 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:38:42,978 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:38:42,978 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-88) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:38:42,980 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-88) [f89ebd8] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 08:38:42,980 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-88) [f89ebd8] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:38:42,987 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-88) [f89ebd8] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:38:45,994 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:38:47,994 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:38:47,994 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:38:47,994 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-89) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:38:47,995 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-89) [18caed52] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 08:38:47,995 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-89) [18caed52] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:38:48,003 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-89) [18caed52] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:38:51,010 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:38:51,996 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:38:51,996 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:38:51,996 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-69) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:38:51,998 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-69) [294797be] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 08:38:51,998 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-69) [294797be] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:38:52,006 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-69) [294797be] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:38:55,015 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:38:57,015 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:38:57,015 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:38:57,015 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-33) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:38:57,017 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-33) [451c3558] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 08:38:57,017 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-33) [451c3558] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:38:57,024 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-33) [451c3558] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:39:00,032 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:39:01,017 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:39:01,017 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:39:01,017 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-61) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:39:01,018 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-61) [20ef7a0c] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 08:39:01,019 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-61) [20ef7a0c] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:39:01,028 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-61) [20ef7a0c] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:39:04,036 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:39:06,037 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:39:06,037 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:39:06,037 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-78) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:39:06,039 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-78) [4e91cc97] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 08:39:06,039 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-78) [4e91cc97] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:39:06,047 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-78) [4e91cc97] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:39:09,055 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:39:10,039 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:39:10,039 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:39:10,039 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-21) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:39:10,041 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-21) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 08:39:10,041 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-21) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:39:10,050 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-21) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:39:13,058 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:39:15,058 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:39:15,058 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:39:15,058 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-62) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:39:15,060 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-62) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 08:39:15,060 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-62) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:39:15,067 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-62) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:39:18,075 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:39:19,060 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:39:19,060 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:39:19,060 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-41) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:39:19,061 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-41) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 08:39:19,061 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-41) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:39:19,070 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-41) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:39:22,078 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:39:24,078 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:39:24,079 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:39:24,080 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-9) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:39:24,081 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-9) [5c6fc8e4] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 08:39:24,081 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-9) [5c6fc8e4] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:39:24,089 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-9) [5c6fc8e4] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:39:27,097 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:39:28,080 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:39:28,080 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:39:28,080 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-64) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:39:28,082 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-64) [785bb27f] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 08:39:28,082 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-64) [785bb27f] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:39:28,089 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-64) [785bb27f] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:39:31,096 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:39:33,096 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:39:33,096 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:39:33,096 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-89) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:39:33,098 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-89) [18caed52] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 08:39:33,098 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-89) [18caed52] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:39:33,106 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-89) [18caed52] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:39:36,113 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:39:37,098 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:39:37,098 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:39:37,098 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-98) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:39:37,100 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-98) [eb8a062] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 08:39:37,100 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-98) [eb8a062] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:39:37,107 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-98) [eb8a062] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:39:40,115 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:39:42,116 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:39:42,117 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:39:42,118 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-33) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:39:42,120 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-33) [451c3558] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 08:39:42,120 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-33) [451c3558] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:39:42,129 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-33) [451c3558] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:39:45,137 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:39:46,118 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:39:46,118 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:39:46,118 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-19) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:39:46,120 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-19) [5a26db10] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 08:39:46,120 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-19) [5a26db10] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:39:46,126 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-19) [5a26db10] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:39:49,132 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:39:51,132 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:39:51,132 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:39:51,132 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-56) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:39:51,134 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-56) [60b15432] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 08:39:51,134 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-56) [60b15432] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:39:51,141 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-56) [60b15432] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:39:54,148 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:39:55,134 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:39:55,134 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:39:55,134 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-21) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:39:55,136 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-21) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 08:39:55,136 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-21) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:39:55,144 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-21) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:39:58,153 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:40:00,153 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:40:00,153 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:40:00,155 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-82) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:40:00,156 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-82) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 08:40:00,156 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-82) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:40:00,163 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-82) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:40:03,170 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:40:04,155 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:40:04,155 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:40:04,155 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-91) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:40:04,157 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-91) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 08:40:04,157 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-91) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:40:04,165 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-91) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:40:04,171 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-91) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:40:07,177 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:40:09,177 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:40:09,177 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:40:09,177 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-76) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:40:09,179 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-76) [1bdd13e4] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 08:40:09,179 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-76) [1bdd13e4] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:40:09,187 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-76) [1bdd13e4] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:40:12,194 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:40:13,179 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:40:13,179 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:40:13,179 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-36) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:40:13,181 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-36) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 08:40:13,181 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-36) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:40:13,189 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-36) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:40:16,197 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:40:18,197 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:40:18,199 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:40:18,200 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-16) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:40:18,201 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-16) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 08:40:18,201 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-16) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:40:18,210 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-16) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:40:21,217 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:40:22,199 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:40:22,199 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:40:22,199 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-52) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:40:22,201 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-52) [7f575cd6] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 08:40:22,201 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-52) [7f575cd6] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:40:22,219 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-52) [7f575cd6] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:40:25,228 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:40:27,228 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:40:27,228 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:40:27,228 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-53) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:40:27,230 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-53) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 08:40:27,230 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-53) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:40:27,239 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-53) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:40:30,249 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:40:31,230 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:40:31,230 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:40:31,230 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-78) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:40:31,232 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-78) [4e91cc97] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 08:40:31,232 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-78) [4e91cc97] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:40:31,240 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-78) [4e91cc97] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:40:34,248 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:40:36,249 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:40:36,249 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:40:36,249 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-14) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:40:36,250 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-14) [6343caea] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 08:40:36,250 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-14) [6343caea] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:40:36,258 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-14) [6343caea] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:40:39,265 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:40:40,251 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:40:40,251 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:40:40,251 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-62) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:40:40,253 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-62) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 08:40:40,253 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-62) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:40:40,262 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-62) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:40:43,270 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:40:45,270 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:40:45,270 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:40:45,270 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-17) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:40:45,272 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-17) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 08:40:45,272 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-17) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:40:45,281 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-17) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:40:48,288 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:40:49,272 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:40:49,272 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:40:49,272 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-88) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:40:49,273 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-88) [f89ebd8] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 08:40:49,273 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-88) [f89ebd8] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:40:49,280 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-88) [f89ebd8] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:40:52,293 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:40:54,293 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:40:54,293 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:40:54,295 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-64) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:40:54,297 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-64) [785bb27f] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 08:40:54,297 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-64) [785bb27f] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:40:54,305 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-64) [785bb27f] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:40:57,313 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:40:58,295 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:40:58,295 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:40:58,295 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-69) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:40:58,297 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-69) [294797be] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 08:40:58,297 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-69) [294797be] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:40:58,304 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-69) [294797be] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:41:01,312 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:41:03,312 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:41:03,312 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:41:03,312 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-87) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:41:03,313 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-87) [1dbf513e] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 08:41:03,313 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-87) [1dbf513e] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:41:03,322 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-87) [1dbf513e] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:41:06,329 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:41:07,314 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:41:07,314 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:41:07,314 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-39) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:41:07,315 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-39) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 08:41:07,315 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-39) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:41:07,322 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-39) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:41:10,328 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:41:12,328 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:41:12,328 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:41:12,328 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-12) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:41:12,330 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-12) [7458e5a2] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 08:41:12,330 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-12) [7458e5a2] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:41:12,337 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-12) [7458e5a2] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:41:15,343 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:41:16,330 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:41:16,330 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:41:16,330 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-8) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:41:16,333 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-8) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 08:41:16,333 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-8) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:41:16,347 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-8) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:41:19,355 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:41:21,355 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:41:21,355 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:41:21,355 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-14) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:41:21,356 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-14) [6343caea] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 08:41:21,356 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-14) [6343caea] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:41:21,364 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-14) [6343caea] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:41:24,371 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:41:25,358 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:41:25,358 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:41:25,358 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-48) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:41:25,360 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-48) [40606675] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 08:41:25,360 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-48) [40606675] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:41:25,370 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-48) [40606675] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:41:28,377 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:41:30,378 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:41:30,378 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:41:30,378 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-99) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:41:30,380 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-99) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 08:41:30,380 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-99) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:41:30,387 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-99) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:41:33,397 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:41:34,380 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:41:34,380 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:41:34,380 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-11) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:41:34,382 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-11) [4df5b8de] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 08:41:34,382 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-11) [4df5b8de] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:41:34,389 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-11) [4df5b8de] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:41:37,396 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:41:39,396 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:41:39,396 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:41:39,396 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-24) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:41:39,398 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-24) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 08:41:39,398 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-24) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:41:39,405 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-24) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:41:42,412 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:41:43,398 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:41:43,398 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:41:43,398 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-32) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:41:43,399 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-32) [56381f88] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 08:41:43,400 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-32) [56381f88] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:41:43,408 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-32) [56381f88] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:41:46,415 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:41:48,416 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:41:48,416 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:41:48,416 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-52) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:41:48,418 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-52) [7f575cd6] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 08:41:48,418 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-52) [7f575cd6] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:41:48,425 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-52) [7f575cd6] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:41:51,431 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:41:52,418 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:41:52,418 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:41:52,418 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-39) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:41:52,420 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-39) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 08:41:52,420 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-39) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:41:52,427 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-39) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:41:55,435 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:41:57,435 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:41:57,435 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:41:57,435 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-12) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:41:57,436 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-12) [7458e5a2] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 08:41:57,436 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-12) [7458e5a2] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:41:57,444 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-12) [7458e5a2] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:42:00,454 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:42:01,437 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:42:01,437 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:42:01,437 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-83) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:42:01,439 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-83) [6d722a1d] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 08:42:01,439 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-83) [6d722a1d] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:42:01,449 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-83) [6d722a1d] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:42:04,457 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:42:06,457 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:42:06,457 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:42:06,457 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-86) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:42:06,459 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-86) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 08:42:06,459 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-86) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:42:06,466 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-86) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:42:09,473 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:42:10,460 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:42:10,460 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:42:10,460 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-20) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:42:10,462 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-20) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 08:42:10,462 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-20) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:42:10,469 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-20) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:42:13,476 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:42:15,476 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:42:15,476 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:42:15,476 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-95) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:42:15,478 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-95) [4bd2250] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 08:42:15,478 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-95) [4bd2250] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:42:15,490 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-95) [4bd2250] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:42:18,499 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:42:19,478 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:42:19,478 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:42:19,478 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-2) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:42:19,480 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-2) [16829b39] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 08:42:19,480 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-2) [16829b39] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:42:19,490 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-2) [16829b39] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:42:22,498 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:42:24,498 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:42:24,498 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:42:24,498 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-24) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:42:24,499 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-24) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 08:42:24,499 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-24) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:42:24,507 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-24) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:42:27,515 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:42:28,500 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:42:28,500 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:42:28,500 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-90) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:42:28,502 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-90) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 08:42:28,502 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-90) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:42:28,508 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-90) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:42:31,516 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:42:33,516 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:42:33,516 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:42:33,516 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-52) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:42:33,517 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-52) [7f575cd6] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 08:42:33,517 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-52) [7f575cd6] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:42:33,525 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-52) [7f575cd6] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:42:36,535 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:42:37,518 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:42:37,518 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:42:37,518 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-84) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:42:37,520 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-84) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 08:42:37,520 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-84) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:42:37,528 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-84) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:42:40,537 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:42:42,537 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:42:42,537 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:42:42,537 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-12) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:42:42,539 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-12) [7458e5a2] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 08:42:42,539 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-12) [7458e5a2] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:42:42,546 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-12) [7458e5a2] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:42:45,554 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:42:46,540 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:42:46,540 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:42:46,540 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-38) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:42:46,542 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-38) [56c37b5] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 08:42:46,542 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-38) [56c37b5] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:42:46,550 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-38) [56c37b5] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:42:49,557 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:42:51,557 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:42:51,557 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:42:51,557 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-41) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:42:51,559 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-41) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 08:42:51,559 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-41) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:42:51,569 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-41) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:42:54,578 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:42:55,559 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:42:55,559 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:42:55,559 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-30) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:42:55,561 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-30) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 08:42:55,561 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-30) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:42:55,569 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-30) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:42:58,578 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:43:00,578 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:43:00,578 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:43:00,578 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-5) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:43:00,579 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-5) [13a89c46] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 08:43:00,579 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-5) [13a89c46] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:43:00,588 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-5) [13a89c46] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:43:03,596 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:43:04,580 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:43:04,580 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:43:04,580 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-65) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:43:04,581 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-65) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 08:43:04,581 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-65) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:43:04,589 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-65) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:43:04,595 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-65) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:43:07,603 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:43:09,603 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:43:09,603 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:43:09,603 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-58) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:43:09,604 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-58) [1a19a2ca] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 08:43:09,604 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-58) [1a19a2ca] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:43:09,612 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-58) [1a19a2ca] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:43:12,619 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:43:13,605 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:43:13,605 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:43:13,605 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-3) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:43:13,607 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-3) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 08:43:13,607 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-3) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:43:13,615 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-3) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:43:16,622 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:43:18,622 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:43:18,622 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:43:18,622 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-53) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:43:18,623 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-53) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 08:43:18,623 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-53) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:43:18,630 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-53) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:43:21,636 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:43:22,624 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:43:22,624 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:43:22,624 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-60) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:43:22,625 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-60) [7bc6f040] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 08:43:22,626 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-60) [7bc6f040] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:43:22,632 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-60) [7bc6f040] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:43:25,642 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:43:27,642 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:43:27,642 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:43:27,642 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-8) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:43:27,644 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-8) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 08:43:27,644 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-8) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:43:27,652 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-8) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:43:30,659 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:43:31,644 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:43:31,644 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:43:31,644 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-34) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:43:31,646 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-34) [3e42dc0b] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 08:43:31,646 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-34) [3e42dc0b] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:43:31,653 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-34) [3e42dc0b] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:43:34,661 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:43:36,661 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:43:36,661 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:43:36,661 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-77) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:43:36,663 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-77) [22f3702] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 08:43:36,663 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-77) [22f3702] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:43:36,671 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-77) [22f3702] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:43:39,680 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:43:40,663 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:43:40,663 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:43:40,663 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-63) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:43:40,664 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-63) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 08:43:40,665 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-63) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:43:40,672 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-63) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:43:43,678 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:43:45,679 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:43:45,679 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:43:45,679 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-74) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:43:45,681 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-74) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 08:43:45,681 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-74) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:43:45,692 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-74) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:43:48,700 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:43:49,681 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:43:49,681 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:43:49,681 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-29) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:43:49,683 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-29) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 08:43:49,683 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-29) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:43:49,691 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-29) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:43:52,698 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:43:54,699 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:43:54,699 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:43:54,701 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-16) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:43:54,703 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-16) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 08:43:54,703 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-16) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:43:54,711 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-16) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:43:57,719 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:43:58,701 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:43:58,701 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:43:58,701 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-52) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:43:58,702 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-52) [7f575cd6] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 08:43:58,702 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-52) [7f575cd6] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:43:58,710 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-52) [7f575cd6] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:44:01,718 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:44:03,718 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:44:03,718 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:44:03,718 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-19) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:44:03,719 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-19) [5a26db10] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 08:44:03,719 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-19) [5a26db10] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:44:03,727 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-19) [5a26db10] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:44:06,734 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:44:07,721 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:44:07,721 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:44:07,721 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-21) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:44:07,723 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-21) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 08:44:07,723 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-21) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:44:07,729 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-21) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:44:10,744 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:44:12,744 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:44:12,744 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:44:12,744 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-8) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:44:12,745 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-8) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 08:44:12,745 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-8) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:44:12,753 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-8) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:44:15,760 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:44:16,747 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:44:16,747 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:44:16,748 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-41) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:44:16,749 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-41) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 08:44:16,750 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-41) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:44:16,760 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-41) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:44:19,767 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:44:21,768 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:44:21,768 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:44:21,768 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-9) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:44:21,769 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-9) [5c6fc8e4] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 08:44:21,769 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-9) [5c6fc8e4] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:44:21,776 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-9) [5c6fc8e4] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:44:24,784 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:44:25,770 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:44:25,770 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:44:25,770 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-5) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:44:25,772 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-5) [13a89c46] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 08:44:25,772 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-5) [13a89c46] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:44:25,779 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-5) [13a89c46] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:44:28,787 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:44:30,787 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:44:30,787 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:44:30,787 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-65) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:44:30,788 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-65) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 08:44:30,788 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-65) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:44:30,796 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-65) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:44:33,806 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:44:34,790 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:44:34,790 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:44:34,790 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-85) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:44:34,792 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-85) [61bce53] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 08:44:34,792 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-85) [61bce53] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:44:34,799 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-85) [61bce53] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:44:37,806 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:44:39,807 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:44:39,807 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:44:39,807 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-3) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:44:39,808 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-3) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 08:44:39,808 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-3) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:44:39,817 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-3) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:44:42,824 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:44:43,809 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:44:43,809 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:44:43,809 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-39) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:44:43,810 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-39) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 08:44:43,810 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-39) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:44:43,817 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-39) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:44:46,825 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:44:48,825 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:44:48,825 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:44:48,825 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-60) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:44:48,826 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-60) [7bc6f040] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 08:44:48,826 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-60) [7bc6f040] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:44:48,834 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-60) [7bc6f040] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:44:51,842 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:44:52,836 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:44:52,836 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:44:52,836 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-21) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:44:52,838 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-21) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 08:44:52,838 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-21) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:44:52,846 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-21) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:44:55,856 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:44:57,856 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:44:57,856 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:44:57,856 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-8) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:44:57,857 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-8) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 08:44:57,857 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-8) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:44:57,865 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-8) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:45:00,873 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:45:01,858 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:45:01,858 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:45:01,858 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-68) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:45:01,859 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-68) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 08:45:01,859 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-68) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:45:01,867 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-68) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:45:04,874 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:45:06,874 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:45:06,874 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:45:06,874 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-47) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:45:06,876 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-47) [4028fe5f] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 08:45:06,876 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-47) [4028fe5f] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:45:06,883 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-47) [4028fe5f] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:45:09,890 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:45:10,876 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:45:10,876 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:45:10,876 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-73) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:45:10,878 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-73) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 08:45:10,878 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-73) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:45:10,886 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-73) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:45:13,895 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:45:15,895 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:45:15,895 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:45:15,895 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-79) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:45:15,897 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-79) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 08:45:15,897 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-79) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:45:15,904 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-79) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:45:18,911 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:45:19,897 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:45:19,897 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:45:19,897 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-61) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:45:19,898 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-61) [20ef7a0c] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 08:45:19,899 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-61) [20ef7a0c] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:45:19,907 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-61) [20ef7a0c] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:45:22,914 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:45:24,914 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:45:24,914 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:45:24,914 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-1) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:45:24,916 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-1) [30db9a86] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 08:45:24,916 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-1) [30db9a86] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:45:24,925 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-1) [30db9a86] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:45:27,933 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:45:28,916 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:45:28,916 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:45:28,916 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-56) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:45:28,918 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-56) [60b15432] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 08:45:28,918 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-56) [60b15432] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:45:28,926 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-56) [60b15432] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:45:31,933 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:45:33,934 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:45:33,934 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:45:33,934 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-12) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:45:33,936 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-12) [7458e5a2] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 08:45:33,936 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-12) [7458e5a2] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:45:33,944 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-12) [7458e5a2] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:45:36,952 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:45:37,936 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:45:37,936 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:45:37,936 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-44) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:45:37,938 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-44) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 08:45:37,938 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-44) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:45:37,945 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-44) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:45:40,955 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:45:42,955 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:45:42,955 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:45:42,955 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-97) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:45:42,956 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-97) [54081188] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 08:45:42,956 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-97) [54081188] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:45:42,964 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-97) [54081188] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:45:45,973 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:45:46,957 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:45:46,957 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:45:46,957 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-13) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:45:46,959 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-13) [7246829] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 08:45:46,959 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-13) [7246829] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:45:46,978 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-13) [7246829] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:45:49,986 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:45:51,986 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:45:51,986 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:45:51,986 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-5) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:45:51,988 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-5) [13a89c46] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 08:45:51,988 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-5) [13a89c46] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:45:51,999 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-5) [13a89c46] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:45:55,007 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:45:55,988 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:45:55,988 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:45:55,988 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-59) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:45:55,990 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-59) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 08:45:55,990 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-59) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:45:55,998 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-59) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:45:59,005 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:46:01,005 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:46:01,005 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:46:01,005 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-87) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:46:01,007 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-87) [1dbf513e] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 08:46:01,007 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-87) [1dbf513e] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:46:01,014 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-87) [1dbf513e] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:46:04,023 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:46:05,007 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:46:05,007 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:46:05,007 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-96) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:46:05,008 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-96) [721bb546] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 08:46:05,009 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-96) [721bb546] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:46:05,014 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-96) [721bb546] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:46:05,019 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-96) [721bb546] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:46:08,026 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:46:10,027 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:46:10,027 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:46:10,027 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-71) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:46:10,030 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-71) [23e814fc] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 08:46:10,030 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-71) [23e814fc] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:46:10,037 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-71) [23e814fc] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:46:13,044 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:46:14,029 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:46:14,029 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:46:14,029 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-15) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:46:14,030 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-15) [3a4dd3fc] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 08:46:14,030 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-15) [3a4dd3fc] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:46:14,038 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-15) [3a4dd3fc] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:46:17,047 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:46:19,047 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:46:19,047 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:46:19,047 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-82) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:46:19,048 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-82) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 08:46:19,048 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-82) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:46:19,055 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-82) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:46:22,064 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:46:23,049 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:46:23,049 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:46:23,049 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-86) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:46:23,051 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-86) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 08:46:23,051 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-86) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:46:23,058 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-86) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:46:26,067 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:46:28,067 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:46:28,067 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:46:28,067 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-99) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:46:28,068 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-99) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 08:46:28,069 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-99) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:46:28,077 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-99) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:46:31,085 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:46:32,069 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:46:32,069 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:46:32,069 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-11) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:46:32,070 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-11) [4df5b8de] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 08:46:32,071 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-11) [4df5b8de] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:46:32,077 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-11) [4df5b8de] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:46:35,085 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:46:37,085 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:46:37,087 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:46:37,087 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-74) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:46:37,088 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-74) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 08:46:37,089 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-74) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:46:37,096 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-74) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:46:40,103 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:46:41,087 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:46:41,087 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:46:41,087 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-75) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:46:41,088 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-75) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 08:46:41,088 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-75) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:46:41,103 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-75) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:46:44,109 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:46:46,110 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:46:46,110 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:46:46,110 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-72) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:46:46,112 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-72) [513300b3] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 08:46:46,112 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-72) [513300b3] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:46:46,120 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-72) [513300b3] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:46:49,129 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:46:50,112 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:46:50,112 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:46:50,112 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-39) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:46:50,115 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-39) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 08:46:50,115 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-39) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:46:50,126 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-39) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:46:53,135 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:46:55,135 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:46:55,135 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:46:55,135 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-19) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:46:55,137 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-19) [5a26db10] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 08:46:55,137 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-19) [5a26db10] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:46:55,144 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-19) [5a26db10] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:46:58,152 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:46:59,137 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:46:59,137 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:46:59,137 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-21) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:46:59,138 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-21) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 08:46:59,139 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-21) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:46:59,146 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-21) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:47:02,154 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:47:04,155 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:47:04,155 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:47:04,155 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-46) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:47:04,160 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-46) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 08:47:04,160 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-46) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:47:04,168 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-46) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:47:07,176 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:47:08,157 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:47:08,157 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:47:08,157 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-68) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:47:08,158 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-68) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 08:47:08,158 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-68) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:47:08,166 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-68) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:47:11,175 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:47:13,175 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:47:13,175 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:47:13,175 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-99) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:47:13,176 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-99) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 08:47:13,176 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-99) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:47:13,185 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-99) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:47:16,192 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:47:17,177 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:47:17,177 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:47:17,177 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-5) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:47:17,178 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-5) [13a89c46] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 08:47:17,179 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-5) [13a89c46] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:47:17,188 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-5) [13a89c46] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:47:20,197 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:47:22,197 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:47:22,197 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:47:22,197 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-24) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:47:22,199 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-24) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 08:47:22,200 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-24) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:47:22,209 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-24) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:47:25,218 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:47:26,199 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:47:26,199 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:47:26,199 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-35) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:47:26,216 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-35) [1ab5e2c1] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 08:47:26,216 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-35) [1ab5e2c1] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:47:26,224 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-35) [1ab5e2c1] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:47:29,232 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:47:31,232 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:47:31,232 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:47:31,232 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-54) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:47:31,234 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-54) [15e2b976] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 08:47:31,234 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-54) [15e2b976] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:47:31,242 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-54) [15e2b976] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:47:34,250 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:47:35,235 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:47:35,235 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:47:35,235 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-53) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:47:35,237 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-53) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 08:47:35,237 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-53) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:47:35,260 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-53) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:47:38,269 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:47:40,269 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:47:40,269 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:47:40,269 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-22) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:47:40,270 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-22) [1e1b6a16] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 08:47:40,270 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-22) [1e1b6a16] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:47:40,279 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-22) [1e1b6a16] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:47:43,286 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:47:44,271 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:47:44,271 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:47:44,271 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-7) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:47:44,272 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-7) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 08:47:44,273 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-7) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:47:44,281 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-7) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:47:47,289 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:47:49,289 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:47:49,289 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:47:49,289 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-86) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:47:49,290 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-86) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 08:47:49,290 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-86) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:47:49,298 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-86) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:47:52,305 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:47:53,291 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:47:53,291 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:47:53,291 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-68) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:47:53,293 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-68) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 08:47:53,293 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-68) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:47:53,300 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-68) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:47:56,309 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:47:58,310 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:47:58,310 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:47:58,310 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-99) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:47:58,311 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-99) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 08:47:58,311 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-99) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:47:58,320 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-99) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:48:01,330 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:48:02,312 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:48:02,312 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:48:02,312 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-31) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:48:02,314 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-31) [2eda49c0] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 08:48:02,314 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-31) [2eda49c0] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:48:02,325 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-31) [2eda49c0] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:48:05,334 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:48:07,334 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:48:07,334 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:48:07,334 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-75) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:48:07,336 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-75) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 08:48:07,336 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-75) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:48:07,347 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-75) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:48:10,357 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:48:11,337 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:48:11,337 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:48:11,337 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-61) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:48:11,339 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-61) [20ef7a0c] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 08:48:11,339 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-61) [20ef7a0c] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:48:11,349 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-61) [20ef7a0c] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:48:14,358 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:48:16,358 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:48:16,358 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:48:16,358 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-40) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:48:16,359 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-40) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 08:48:16,360 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-40) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:48:16,367 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-40) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:48:19,374 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:48:20,360 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:48:20,360 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:48:20,360 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-56) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:48:20,361 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-56) [60b15432] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 08:48:20,361 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-56) [60b15432] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:48:20,370 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-56) [60b15432] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:48:23,378 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:48:25,378 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:48:25,378 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:48:25,378 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-22) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:48:25,379 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-22) [1e1b6a16] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 08:48:25,379 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-22) [1e1b6a16] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:48:25,387 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-22) [1e1b6a16] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:48:28,394 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:48:29,380 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:48:29,380 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:48:29,380 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-44) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:48:29,382 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-44) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 08:48:29,382 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-44) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:48:29,389 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-44) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:48:32,397 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:48:34,397 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:48:34,397 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:48:34,399 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-86) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:48:34,400 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-86) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 08:48:34,400 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-86) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:48:34,407 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-86) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:48:37,415 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:48:38,399 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:48:38,399 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:48:38,399 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-13) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:48:38,400 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-13) [7246829] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 08:48:38,401 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-13) [7246829] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:48:38,408 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-13) [7246829] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:48:41,418 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:48:43,418 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:48:43,418 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:48:43,418 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-23) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:48:43,419 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-23) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 08:48:43,420 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-23) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:48:43,427 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-23) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:48:46,434 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:48:47,420 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:48:47,420 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:48:47,420 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-29) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:48:47,422 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-29) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 08:48:47,422 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-29) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:48:47,429 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-29) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:48:50,436 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:48:52,436 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:48:52,436 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:48:52,436 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-79) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:48:52,437 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-79) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 08:48:52,438 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-79) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:48:52,446 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-79) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:48:55,454 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:48:56,438 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:48:56,438 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:48:56,438 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-37) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:48:56,439 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-37) [310dcfa0] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 08:48:56,439 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-37) [310dcfa0] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:48:56,447 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-37) [310dcfa0] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:48:59,455 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:49:01,455 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:49:01,455 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:49:01,455 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-80) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:49:01,457 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-80) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 08:49:01,457 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-80) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:49:01,466 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-80) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:49:04,476 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:49:05,457 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:49:05,457 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:49:05,457 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-45) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:49:05,459 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-45) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 08:49:05,460 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-45) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:49:05,468 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-45) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:49:05,475 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-45) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:49:08,483 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:49:10,483 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:49:10,483 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:49:10,483 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-38) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:49:10,484 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-38) [56c37b5] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 08:49:10,484 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-38) [56c37b5] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:49:10,492 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-38) [56c37b5] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:49:13,500 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:49:14,485 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:49:14,485 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:49:14,485 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-50) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:49:14,486 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-50) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 08:49:14,486 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-50) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:49:14,493 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-50) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:49:17,501 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:49:19,501 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:49:19,501 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:49:19,501 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-27) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:49:19,502 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-27) [3ede3f2d] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 08:49:19,502 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-27) [3ede3f2d] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:49:19,510 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-27) [3ede3f2d] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:49:22,518 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:49:23,503 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:49:23,503 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:49:23,503 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-11) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:49:23,504 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-11) [4df5b8de] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 08:49:23,504 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-11) [4df5b8de] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:49:23,512 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-11) [4df5b8de] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:49:26,530 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:49:28,530 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:49:28,530 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:49:28,530 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-69) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:49:28,531 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-69) [294797be] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 08:49:28,531 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-69) [294797be] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:49:28,540 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-69) [294797be] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:49:31,550 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:49:32,533 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:49:32,533 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:49:32,533 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-6) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:49:32,535 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-6) [23c78bb] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 08:49:32,535 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-6) [23c78bb] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:49:32,542 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-6) [23c78bb] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:49:35,552 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:49:37,552 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:49:37,552 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:49:37,552 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-57) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:49:37,554 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-57) [16b48795] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 08:49:37,554 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-57) [16b48795] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:49:37,561 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-57) [16b48795] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:49:40,569 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:49:41,554 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:49:41,554 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:49:41,554 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-39) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:49:41,555 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-39) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 08:49:41,556 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-39) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:49:41,563 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-39) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:49:44,570 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:49:46,571 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:49:46,571 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:49:46,571 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-53) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:49:46,572 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-53) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 08:49:46,573 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-53) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:49:46,579 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-53) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:49:49,586 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:49:50,572 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:49:50,572 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:49:50,572 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-21) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:49:50,574 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-21) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 08:49:50,574 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-21) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:49:50,582 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-21) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:49:53,589 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:49:55,589 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:49:55,589 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:49:55,589 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-7) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:49:55,590 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-7) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 08:49:55,590 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-7) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:49:55,597 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-7) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:49:58,605 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:49:59,591 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:49:59,591 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:49:59,591 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-97) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:49:59,593 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-97) [54081188] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 08:49:59,593 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-97) [54081188] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:49:59,600 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-97) [54081188] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:50:02,607 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:50:04,607 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:50:04,607 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:50:04,607 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-95) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:50:04,609 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-95) [4bd2250] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 08:50:04,609 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-95) [4bd2250] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:50:04,618 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-95) [4bd2250] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:50:07,625 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:50:08,609 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:50:08,609 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:50:08,609 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-18) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:50:08,612 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-18) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 08:50:08,612 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-18) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:50:08,622 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-18) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:50:11,631 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:50:13,631 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:50:13,631 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:50:13,631 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-85) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:50:13,632 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-85) [61bce53] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 08:50:13,632 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-85) [61bce53] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:50:13,640 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-85) [61bce53] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:50:16,648 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:50:17,633 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:50:17,633 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:50:17,633 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-61) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:50:17,635 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-61) [20ef7a0c] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 08:50:17,636 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-61) [20ef7a0c] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:50:17,645 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-61) [20ef7a0c] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:50:20,653 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:50:22,654 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:50:22,654 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:50:22,654 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-10) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:50:22,655 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-10) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 08:50:22,655 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-10) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:50:22,664 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-10) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:50:25,672 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:50:26,656 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:50:26,656 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:50:26,656 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-56) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:50:26,658 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-56) [60b15432] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 08:50:26,658 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-56) [60b15432] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:50:26,666 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-56) [60b15432] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:50:29,674 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:50:31,674 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:50:31,674 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:50:31,674 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-45) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:50:31,675 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-45) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 08:50:31,675 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-45) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:50:31,683 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-45) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:50:34,691 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:50:35,676 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:50:35,676 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:50:35,676 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-44) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:50:35,678 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-44) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 08:50:35,678 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-44) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:50:35,687 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-44) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:50:38,695 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:50:40,706 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:50:40,706 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:50:40,706 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-28) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:50:40,707 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-28) [60adf93f] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 08:50:40,708 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-28) [60adf93f] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:50:40,714 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-28) [60adf93f] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:50:43,721 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:50:44,708 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:50:44,708 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:50:44,708 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-27) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:50:44,710 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-27) [3ede3f2d] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 08:50:44,710 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-27) [3ede3f2d] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:50:44,717 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-27) [3ede3f2d] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:50:47,725 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:50:49,726 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:50:49,726 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:50:49,726 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-23) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:50:49,727 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-23) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 08:50:49,727 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-23) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:50:49,735 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-23) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:50:52,744 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:50:53,727 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:50:53,727 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:50:53,727 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-18) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:50:53,729 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-18) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 08:50:53,729 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-18) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:50:53,738 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-18) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:50:56,746 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:50:58,746 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:50:58,746 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:50:58,746 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-90) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:50:58,747 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-90) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 08:50:58,748 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-90) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:50:58,755 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-90) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:51:01,765 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:51:02,750 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:51:02,750 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:51:02,750 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-61) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:51:02,751 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-61) [20ef7a0c] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 08:51:02,751 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-61) [20ef7a0c] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:51:02,759 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-61) [20ef7a0c] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:51:05,767 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:51:07,767 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:51:07,767 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:51:07,767 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-71) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:51:07,768 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-71) [23e814fc] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 08:51:07,769 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-71) [23e814fc] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:51:07,777 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-71) [23e814fc] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:51:10,785 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:51:11,769 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:51:11,769 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:51:11,769 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-12) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:51:11,770 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-12) [7458e5a2] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 08:51:11,771 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-12) [7458e5a2] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:51:11,778 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-12) [7458e5a2] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:51:14,786 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:51:16,786 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:51:16,786 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:51:16,786 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-94) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:51:16,787 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-94) [5dd0314d] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 08:51:16,787 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-94) [5dd0314d] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:51:16,795 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-94) [5dd0314d] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:51:19,804 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:51:20,788 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:51:20,788 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:51:20,788 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-50) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:51:20,789 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-50) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 08:51:20,789 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-50) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:51:20,797 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-50) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:51:23,804 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:51:25,804 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:51:25,804 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:51:25,804 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-88) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:51:25,805 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-88) [f89ebd8] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 08:51:25,805 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-88) [f89ebd8] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:51:25,812 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-88) [f89ebd8] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:51:28,819 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:51:29,806 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:51:29,806 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:51:29,806 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-47) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:51:29,808 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-47) [4028fe5f] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 08:51:29,808 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-47) [4028fe5f] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:51:29,814 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-47) [4028fe5f] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:51:32,821 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:51:34,822 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:51:34,822 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:51:34,822 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-5) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:51:34,823 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-5) [13a89c46] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 08:51:34,823 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-5) [13a89c46] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:51:34,830 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-5) [13a89c46] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:51:37,839 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:51:38,824 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:51:38,824 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:51:38,824 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-16) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:51:38,826 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-16) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 08:51:38,826 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-16) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:51:38,834 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-16) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:51:41,842 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:51:43,842 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:51:43,842 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:51:43,844 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-72) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:51:43,846 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-72) [513300b3] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 08:51:43,846 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-72) [513300b3] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:51:43,854 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-72) [513300b3] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:51:46,860 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:51:47,844 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:51:47,844 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:51:47,844 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-78) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:51:47,846 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-78) [4e91cc97] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 08:51:47,846 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-78) [4e91cc97] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:51:47,853 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-78) [4e91cc97] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:51:50,860 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:51:52,860 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:51:52,860 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:51:52,860 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-19) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:51:52,861 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-19) [5a26db10] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 08:51:52,862 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-19) [5a26db10] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:51:52,871 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-19) [5a26db10] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:51:55,878 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:51:56,862 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:51:56,862 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:51:56,863 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-100) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:51:56,865 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-100) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 08:51:56,865 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-100) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:51:56,874 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-100) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:51:59,883 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:52:01,883 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:52:01,885 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:52:01,885 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-46) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:52:01,887 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-46) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 08:52:01,888 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-46) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:52:01,898 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-46) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:52:04,910 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:52:05,885 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:52:05,885 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:52:05,885 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-68) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:52:05,887 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-68) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 08:52:05,887 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-68) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:52:05,901 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-68) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:52:05,915 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-68) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:52:08,924 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:52:10,924 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:52:10,924 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:52:10,924 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-92) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:52:10,925 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-92) [3c53e142] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 08:52:10,925 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-92) [3c53e142] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:52:10,933 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-92) [3c53e142] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:52:13,940 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:52:14,926 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:52:14,926 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:52:14,926 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-36) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:52:14,927 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-36) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 08:52:14,927 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-36) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:52:14,934 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-36) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:52:17,941 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:52:19,942 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:52:19,942 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:52:19,942 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-29) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:52:19,943 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-29) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 08:52:19,944 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-29) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:52:19,951 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-29) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:52:22,958 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:52:23,943 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:52:23,943 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:52:23,943 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-35) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:52:23,945 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-35) [1ab5e2c1] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 08:52:23,945 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-35) [1ab5e2c1] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:52:23,952 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-35) [1ab5e2c1] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:52:26,960 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:52:28,960 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:52:28,960 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:52:28,960 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-61) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:52:28,961 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-61) [20ef7a0c] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 08:52:28,961 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-61) [20ef7a0c] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:52:28,968 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-61) [20ef7a0c] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:52:31,976 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:52:32,962 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:52:32,962 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:52:32,962 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-80) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:52:32,963 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-80) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 08:52:32,963 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-80) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:52:32,970 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-80) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:52:35,978 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:52:37,978 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:52:37,978 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:52:37,980 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-43) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:52:37,983 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-43) [2d3b2827] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 08:52:37,983 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-43) [2d3b2827] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:52:37,993 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-43) [2d3b2827] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:52:41,002 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:52:41,980 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:52:41,980 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:52:41,980 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-82) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:52:41,982 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-82) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 08:52:41,982 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-82) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:52:41,992 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-82) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:52:44,999 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:52:47,000 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:52:47,000 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:52:47,000 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-41) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:52:47,002 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-41) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 08:52:47,002 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-41) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:52:47,011 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-41) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:52:50,019 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:52:51,002 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:52:51,002 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:52:51,002 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-9) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:52:51,004 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-9) [5c6fc8e4] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 08:52:51,004 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-9) [5c6fc8e4] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:52:51,012 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-9) [5c6fc8e4] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:52:54,020 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:52:56,020 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:52:56,020 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:52:56,022 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-11) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:52:56,024 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-11) [4df5b8de] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 08:52:56,024 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-11) [4df5b8de] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:52:56,031 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-11) [4df5b8de] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:52:59,038 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:53:00,022 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:53:00,022 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:53:00,022 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-74) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:53:00,023 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-74) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 08:53:00,024 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-74) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:53:00,031 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-74) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:53:03,037 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:53:05,037 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:53:05,037 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:53:05,037 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-6) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:53:05,038 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-6) [23c78bb] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 08:53:05,038 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-6) [23c78bb] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:53:05,046 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-6) [23c78bb] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:53:08,054 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:53:09,039 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:53:09,039 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:53:09,039 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-57) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:53:09,041 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-57) [16b48795] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 08:53:09,041 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-57) [16b48795] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:53:09,048 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-57) [16b48795] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:53:12,056 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:53:14,056 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:53:14,056 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:53:14,056 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-39) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:53:14,058 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-39) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 08:53:14,058 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-39) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:53:14,065 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-39) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:53:17,072 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:53:18,058 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:53:18,058 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:53:18,058 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-53) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:53:18,059 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-53) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 08:53:18,059 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-53) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:53:18,068 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-53) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:53:21,074 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:53:23,075 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:53:23,075 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:53:23,075 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-62) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:53:23,077 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-62) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 08:53:23,077 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-62) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:53:23,085 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-62) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:53:26,093 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:53:27,077 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:53:27,077 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:53:27,077 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-8) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:53:27,080 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-8) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 08:53:27,080 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-8) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:53:27,088 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-8) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:53:30,095 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:53:32,095 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:53:32,095 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:53:32,095 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-86) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:53:32,097 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-86) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 08:53:32,097 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-86) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:53:32,104 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-86) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:53:35,114 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:53:36,098 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:53:36,098 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:53:36,098 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-93) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:53:36,100 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-93) [4f020087] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 08:53:36,100 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-93) [4f020087] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:53:36,109 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-93) [4f020087] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:53:39,118 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:53:41,118 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:53:41,118 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:53:41,118 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-89) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:53:41,120 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-89) [18caed52] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 08:53:41,120 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-89) [18caed52] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:53:41,128 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-89) [18caed52] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:53:44,135 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:53:45,120 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:53:45,120 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:53:45,120 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-24) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:53:45,122 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-24) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 08:53:45,122 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-24) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:53:45,130 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-24) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:53:48,138 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:53:50,138 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:53:50,138 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:53:50,138 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-35) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:53:50,140 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-35) [1ab5e2c1] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 08:53:50,140 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-35) [1ab5e2c1] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:53:50,148 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-35) [1ab5e2c1] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:53:53,155 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:53:54,140 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:53:54,140 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:53:54,140 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-96) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:53:54,141 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-96) [721bb546] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 08:53:54,142 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-96) [721bb546] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:53:54,149 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-96) [721bb546] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:53:57,165 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:53:59,166 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:53:59,166 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:53:59,166 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-39) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:53:59,167 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-39) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 08:53:59,167 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-39) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:53:59,174 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-39) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:54:02,184 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:54:03,168 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:54:03,168 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:54:03,168 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-21) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:54:03,170 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-21) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 08:54:03,170 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-21) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:54:03,180 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-21) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:54:06,188 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:54:08,188 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:54:08,188 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:54:08,188 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-34) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:54:08,190 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-34) [3e42dc0b] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 08:54:08,190 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-34) [3e42dc0b] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:54:08,201 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-34) [3e42dc0b] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:54:11,211 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:54:12,190 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:54:12,190 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:54:12,190 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-50) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:54:12,192 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-50) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 08:54:12,192 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-50) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:54:12,199 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-50) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:54:15,205 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:54:17,206 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:54:17,206 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:54:17,206 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-13) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:54:17,208 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-13) [7246829] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 08:54:17,208 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-13) [7246829] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:54:17,219 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-13) [7246829] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:54:20,229 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:54:21,208 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:54:21,208 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:54:21,208 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-47) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:54:21,211 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-47) [4028fe5f] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 08:54:21,211 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-47) [4028fe5f] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:54:21,218 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-47) [4028fe5f] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:54:24,224 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:54:26,225 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:54:26,225 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:54:26,225 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-36) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:54:26,227 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-36) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 08:54:26,227 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-36) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:54:26,235 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-36) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:54:29,244 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:54:30,226 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:54:30,226 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:54:30,226 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-16) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:54:30,228 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-16) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 08:54:30,228 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-16) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:54:30,236 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-16) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:54:33,243 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:54:35,244 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:54:35,244 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:54:35,244 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-90) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:54:35,245 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-90) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 08:54:35,245 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-90) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:54:35,254 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-90) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:54:38,261 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:54:39,246 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:54:39,246 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:54:39,246 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-78) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:54:39,247 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-78) [4e91cc97] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 08:54:39,247 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-78) [4e91cc97] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:54:39,255 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-78) [4e91cc97] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:54:42,265 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:54:44,265 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:54:44,265 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:54:44,267 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-71) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:54:44,268 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-71) [23e814fc] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 08:54:44,268 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-71) [23e814fc] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:54:44,275 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-71) [23e814fc] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:54:47,282 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:54:48,268 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:54:48,268 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:54:48,268 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-100) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:54:48,270 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-100) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 08:54:48,270 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-100) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:54:48,278 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-100) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:54:51,284 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:54:53,285 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:54:53,285 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:54:53,285 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-44) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:54:53,286 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-44) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 08:54:53,287 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-44) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:54:53,294 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-44) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:54:56,302 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:54:57,287 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:54:57,287 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:54:57,287 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-28) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:54:57,289 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-28) [60adf93f] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 08:54:57,289 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-28) [60adf93f] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:54:57,300 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-28) [60adf93f] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:55:00,308 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:55:02,308 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:55:02,308 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:55:02,308 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-88) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:55:02,310 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-88) [f89ebd8] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 08:55:02,310 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-88) [f89ebd8] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:55:02,319 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-88) [f89ebd8] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:55:05,328 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:55:06,311 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:55:06,311 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:55:06,311 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-32) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:55:06,312 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-32) [56381f88] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 08:55:06,312 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-32) [56381f88] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:55:06,320 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-32) [56381f88] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:55:06,325 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-32) [56381f88] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:55:09,332 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:55:11,332 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:55:11,332 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:55:11,332 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-85) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:55:11,334 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-85) [61bce53] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 08:55:11,334 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-85) [61bce53] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:55:11,341 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-85) [61bce53] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:55:14,349 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:55:15,334 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:55:15,334 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:55:15,334 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-72) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:55:15,335 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-72) [513300b3] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 08:55:15,335 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-72) [513300b3] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:55:15,344 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-72) [513300b3] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:55:18,352 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:55:20,353 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:55:20,353 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:55:20,353 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-40) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:55:20,355 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-40) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 08:55:20,355 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-40) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:55:20,362 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-40) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:55:23,370 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:55:24,355 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:55:24,355 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:55:24,355 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-67) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:55:24,356 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-67) [6cea7f79] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 08:55:24,357 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-67) [6cea7f79] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:55:24,364 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-67) [6cea7f79] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:55:27,373 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:55:29,373 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:55:29,373 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:55:29,373 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-22) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:55:29,374 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-22) [1e1b6a16] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 08:55:29,374 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-22) [1e1b6a16] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:55:29,382 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-22) [1e1b6a16] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:55:32,390 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:55:33,375 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:55:33,375 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:55:33,375 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-7) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:55:33,377 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-7) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 08:55:33,377 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-7) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:55:33,384 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-7) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:55:36,393 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:55:38,393 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:55:38,393 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:55:38,393 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-50) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:55:38,395 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-50) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 08:55:38,395 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-50) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:55:38,403 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-50) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:55:41,410 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:55:42,395 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:55:42,395 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:55:42,396 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-27) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:55:42,398 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-27) [3ede3f2d] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 08:55:42,398 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-27) [3ede3f2d] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:55:42,408 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-27) [3ede3f2d] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:55:45,416 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:55:47,416 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:55:47,416 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:55:47,416 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-23) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:55:47,418 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-23) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 08:55:47,418 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-23) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:55:47,426 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-23) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:55:50,435 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:55:51,418 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:55:51,418 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:55:51,418 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-59) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:55:51,420 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-59) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 08:55:51,420 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-59) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:55:51,428 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-59) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:55:54,436 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:55:56,436 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:55:56,436 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:55:56,436 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-79) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:55:56,444 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-79) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 08:55:56,444 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-79) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:55:56,451 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-79) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:55:59,459 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:56:00,438 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:56:00,438 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:56:00,438 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-90) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:56:00,439 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-90) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 08:56:00,440 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-90) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:56:00,447 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-90) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:56:03,455 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:56:05,455 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:56:05,455 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:56:05,455 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-78) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:56:05,456 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-78) [4e91cc97] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 08:56:05,456 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-78) [4e91cc97] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:56:05,464 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-78) [4e91cc97] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:56:08,470 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:56:09,457 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:56:09,457 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:56:09,457 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-4) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:56:09,459 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-4) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 08:56:09,459 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-4) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:56:09,468 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-4) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:56:12,477 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:56:14,477 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:56:14,479 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:56:14,479 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-38) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:56:14,480 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-38) [56c37b5] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 08:56:14,481 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-38) [56c37b5] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:56:14,488 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-38) [56c37b5] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:56:17,497 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:56:18,479 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:56:18,479 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:56:18,479 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-97) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:56:18,481 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-97) [54081188] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 08:56:18,481 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-97) [54081188] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:56:18,488 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-97) [54081188] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:56:21,494 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:56:23,494 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:56:23,494 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:56:23,494 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-68) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:56:23,496 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-68) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 08:56:23,496 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-68) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:56:23,503 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-68) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:56:26,510 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:56:27,496 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:56:27,496 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:56:27,496 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-95) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:56:27,498 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-95) [4bd2250] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 08:56:27,498 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-95) [4bd2250] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:56:27,505 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-95) [4bd2250] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:56:30,514 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:56:32,514 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:56:32,514 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:56:32,514 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-69) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:56:32,515 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-69) [294797be] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 08:56:32,516 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-69) [294797be] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:56:32,523 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-69) [294797be] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:56:35,533 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:56:36,516 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:56:36,516 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:56:36,516 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-16) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:56:36,517 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-16) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 08:56:36,517 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-16) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:56:36,525 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-16) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:56:39,533 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:56:41,534 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:56:41,534 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:56:41,534 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-54) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:56:41,536 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-54) [15e2b976] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 08:56:41,536 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-54) [15e2b976] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:56:41,543 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-54) [15e2b976] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:56:44,552 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:56:45,536 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:56:45,536 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:56:45,536 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-10) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:56:45,538 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-10) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 08:56:45,538 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-10) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:56:45,545 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-10) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:56:48,554 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:56:50,555 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:56:50,555 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:56:50,555 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-78) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:56:50,556 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-78) [4e91cc97] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 08:56:50,556 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-78) [4e91cc97] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:56:50,565 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-78) [4e91cc97] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:56:53,572 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:56:54,557 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:56:54,557 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:56:54,557 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-12) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:56:54,558 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-12) [7458e5a2] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 08:56:54,559 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-12) [7458e5a2] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:56:54,566 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-12) [7458e5a2] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:56:57,574 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:56:59,574 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:56:59,574 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:56:59,574 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-38) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:56:59,575 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-38) [56c37b5] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 08:56:59,575 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-38) [56c37b5] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:56:59,583 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-38) [56c37b5] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:57:02,592 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:57:03,576 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:57:03,576 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:57:03,576 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-97) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:57:03,578 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-97) [54081188] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 08:57:03,578 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-97) [54081188] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:57:03,589 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-97) [54081188] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:57:06,597 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:57:08,597 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:57:08,597 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:57:08,597 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-27) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:57:08,599 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-27) [3ede3f2d] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 08:57:08,599 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-27) [3ede3f2d] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:57:08,608 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-27) [3ede3f2d] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:57:11,615 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:57:12,599 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:57:12,599 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:57:12,599 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-89) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:57:12,601 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-89) [18caed52] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 08:57:12,601 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-89) [18caed52] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:57:12,611 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-89) [18caed52] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:57:15,620 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:57:17,620 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:57:17,620 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:57:17,620 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-58) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:57:17,621 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-58) [1a19a2ca] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 08:57:17,621 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-58) [1a19a2ca] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:57:17,629 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-58) [1a19a2ca] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:57:20,637 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:57:21,622 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:57:21,622 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:57:21,622 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-33) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:57:21,624 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-33) [451c3558] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 08:57:21,624 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-33) [451c3558] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:57:21,634 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-33) [451c3558] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:57:24,643 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:57:26,643 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:57:26,643 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:57:26,643 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-54) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:57:26,644 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-54) [15e2b976] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 08:57:26,644 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-54) [15e2b976] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:57:26,653 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-54) [15e2b976] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:57:29,662 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:57:30,645 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:57:30,645 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:57:30,645 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-83) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:57:30,646 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-83) [6d722a1d] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 08:57:30,646 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-83) [6d722a1d] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:57:30,653 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-83) [6d722a1d] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:57:33,661 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:57:35,661 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:57:35,661 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:57:35,661 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-15) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:57:35,662 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-15) [3a4dd3fc] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 08:57:35,662 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-15) [3a4dd3fc] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:57:35,670 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-15) [3a4dd3fc] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:57:38,677 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:57:39,664 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:57:39,664 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:57:39,664 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-94) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:57:39,665 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-94) [5dd0314d] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 08:57:39,665 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-94) [5dd0314d] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:57:39,673 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-94) [5dd0314d] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:57:42,681 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:57:44,681 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:57:44,681 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:57:44,682 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-20) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:57:44,684 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-20) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 08:57:44,684 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-20) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:57:44,691 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-20) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:57:47,699 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:57:48,683 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:57:48,683 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:57:48,683 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-63) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:57:48,684 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-63) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 08:57:48,684 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-63) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:57:48,693 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-63) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:57:51,703 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:57:53,703 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:57:53,703 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:57:53,703 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-99) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:57:53,704 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-99) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 08:57:53,704 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-99) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:57:53,712 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-99) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:57:56,719 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:57:57,705 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:57:57,705 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:57:57,705 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-18) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:57:57,707 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-18) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 08:57:57,707 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-18) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:57:57,714 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-18) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:58:00,722 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:58:02,722 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:58:02,722 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:58:02,722 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-29) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:58:02,723 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-29) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 08:58:02,723 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-29) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:58:02,732 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-29) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:58:05,742 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:58:06,724 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:58:06,724 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:58:06,724 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-72) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:58:06,726 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-72) [513300b3] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 08:58:06,726 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-72) [513300b3] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:58:06,733 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-72) [513300b3] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:58:06,737 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-72) [513300b3] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:58:09,744 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:58:11,744 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:58:11,744 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:58:11,744 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-40) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:58:11,745 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-40) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 08:58:11,745 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-40) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:58:11,753 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-40) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:58:14,759 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:58:15,746 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:58:15,746 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:58:15,746 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-78) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:58:15,748 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-78) [4e91cc97] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 08:58:15,748 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-78) [4e91cc97] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:58:15,755 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-78) [4e91cc97] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:58:18,762 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:58:20,763 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:58:20,763 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:58:20,763 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-14) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:58:20,764 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-14) [6343caea] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 08:58:20,765 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-14) [6343caea] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:58:20,773 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-14) [6343caea] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:58:23,780 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:58:24,764 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:58:24,764 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:58:24,766 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-48) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:58:24,768 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-48) [40606675] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 08:58:24,768 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-48) [40606675] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:58:24,775 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-48) [40606675] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:58:27,784 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:58:29,784 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:58:29,784 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:58:29,786 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-97) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:58:29,787 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-97) [54081188] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 08:58:29,787 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-97) [54081188] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:58:29,796 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-97) [54081188] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:58:32,804 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:58:33,786 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:58:33,786 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:58:33,786 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-64) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:58:33,787 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-64) [785bb27f] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 08:58:33,788 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-64) [785bb27f] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:58:33,795 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-64) [785bb27f] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:58:36,803 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:58:38,803 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:58:38,803 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:58:38,803 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-73) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:58:38,804 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-73) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 08:58:38,804 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-73) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:58:38,812 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-73) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:58:41,819 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:58:42,805 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:58:42,805 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:58:42,805 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-98) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:58:42,807 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-98) [eb8a062] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 08:58:42,807 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-98) [eb8a062] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:58:42,813 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-98) [eb8a062] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:58:45,821 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:58:47,821 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:58:47,821 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:58:47,821 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-3) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:58:47,823 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-3) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 08:58:47,824 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-3) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:58:47,833 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-3) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:58:50,841 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:58:51,823 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:58:51,823 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:58:51,823 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-66) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:58:51,825 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-66) [e582aa6] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 08:58:51,825 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-66) [e582aa6] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:58:51,831 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-66) [e582aa6] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:58:54,839 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:58:56,839 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:58:56,839 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:58:56,839 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-80) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:58:56,841 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-80) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 08:58:56,841 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-80) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:58:56,848 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-80) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:58:59,855 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:59:00,841 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:59:00,841 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:59:00,841 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-42) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:59:00,843 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-42) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 08:59:00,843 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-42) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:59:00,849 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-42) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:59:03,856 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:59:05,856 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:59:05,856 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:59:05,858 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-22) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:59:05,859 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-22) [1e1b6a16] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 08:59:05,859 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-22) [1e1b6a16] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:59:05,865 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-22) [1e1b6a16] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:59:08,874 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:59:09,858 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:59:09,858 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:59:09,858 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-77) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:59:09,860 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-77) [22f3702] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 08:59:09,860 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-77) [22f3702] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:59:09,866 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-77) [22f3702] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:59:12,874 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:59:14,875 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:59:14,875 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:59:14,875 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-9) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:59:14,876 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-9) [5c6fc8e4] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 08:59:14,877 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-9) [5c6fc8e4] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:59:14,884 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-9) [5c6fc8e4] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:59:17,890 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:59:18,876 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:59:18,876 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:59:18,876 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-11) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:59:18,877 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-11) [4df5b8de] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 08:59:18,877 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-11) [4df5b8de] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:59:18,885 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-11) [4df5b8de] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:59:21,891 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:59:23,891 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:59:23,891 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:59:23,893 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-5) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:59:23,895 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-5) [13a89c46] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 08:59:23,895 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-5) [13a89c46] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:59:23,903 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-5) [13a89c46] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:59:26,912 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:59:27,893 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:59:27,893 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:59:27,893 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-81) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:59:27,895 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-81) [2dc3b30d] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 08:59:27,895 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-81) [2dc3b30d] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:59:27,903 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-81) [2dc3b30d] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:59:30,911 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:59:32,911 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:59:32,911 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:59:32,911 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-3) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:59:32,913 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-3) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 08:59:32,913 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-3) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:59:32,919 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-3) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:59:35,929 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:59:36,913 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:59:36,913 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:59:36,913 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-10) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:59:36,915 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-10) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 08:59:36,915 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-10) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:59:36,923 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-10) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:59:39,931 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:59:41,931 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:59:41,931 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:59:41,931 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-67) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:59:41,932 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-67) [6cea7f79] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 08:59:41,933 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-67) [6cea7f79] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:59:41,940 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-67) [6cea7f79] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:59:44,948 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:59:45,934 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:59:45,934 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:59:45,934 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-62) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:59:45,936 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-62) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 08:59:45,936 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-62) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:59:45,943 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-62) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:59:48,950 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:59:50,951 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:59:50,951 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:59:50,951 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-44) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:59:50,953 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-44) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 08:59:50,953 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-44) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:59:50,960 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-44) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:59:53,967 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:59:54,953 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:59:54,953 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:59:54,953 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-30) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:59:54,955 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-30) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 08:59:54,955 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-30) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 08:59:54,962 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-30) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 08:59:57,971 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 08:59:59,971 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:00:00,007 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:00:00,008 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-9) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:00:00,019 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-9) [5c6fc8e4] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:00:00,020 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-9) [5c6fc8e4] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:00:00,027 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-9) [5c6fc8e4] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:00:02,212 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-23) Attempting to update VMs/Templates Ovf. >2014-10-08 09:00:03,037 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:00:03,973 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:00:03,973 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:00:03,973 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-18) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:00:03,975 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-18) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:00:03,975 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-18) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:00:03,984 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-18) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:00:06,991 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:00:08,994 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:00:08,994 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:00:08,994 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-81) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:00:08,996 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-81) [2dc3b30d] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:00:08,996 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-81) [2dc3b30d] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:00:09,003 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-81) [2dc3b30d] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:00:12,010 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:00:12,996 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:00:12,996 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:00:12,996 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-61) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:00:12,998 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-61) [20ef7a0c] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:00:12,998 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-61) [20ef7a0c] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:00:13,006 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-61) [20ef7a0c] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:00:16,012 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:00:18,013 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:00:18,013 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:00:18,013 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-83) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:00:18,015 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-83) [6d722a1d] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:00:18,015 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-83) [6d722a1d] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:00:18,023 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-83) [6d722a1d] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:00:21,031 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:00:22,014 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:00:22,014 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:00:22,014 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-78) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:00:22,016 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-78) [4e91cc97] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:00:22,016 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-78) [4e91cc97] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:00:22,027 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-78) [4e91cc97] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:00:25,035 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:00:27,036 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:00:27,036 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:00:27,036 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-82) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:00:27,038 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-82) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:00:27,038 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-82) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:00:27,045 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-82) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:00:30,052 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:00:31,038 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:00:31,038 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:00:31,038 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-44) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:00:31,040 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-44) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:00:31,040 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-44) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:00:31,046 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-44) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:00:34,054 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:00:36,055 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:00:36,055 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:00:36,056 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-97) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:00:36,058 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-97) [54081188] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:00:36,058 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-97) [54081188] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:00:36,067 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-97) [54081188] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:00:39,074 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:00:40,057 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:00:40,057 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:00:40,057 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-9) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:00:40,058 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-9) [5c6fc8e4] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:00:40,059 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-9) [5c6fc8e4] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:00:40,066 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-9) [5c6fc8e4] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:00:43,075 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:00:45,076 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:00:45,076 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:00:45,076 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-23) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:00:45,077 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-23) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:00:45,078 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-23) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:00:45,085 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-23) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:00:48,091 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:00:49,078 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:00:49,078 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:00:49,078 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-75) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:00:49,079 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-75) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:00:49,079 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-75) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:00:49,087 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-75) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:00:52,095 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:00:54,095 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:00:54,095 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:00:54,095 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-33) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:00:54,113 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-33) [451c3558] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:00:54,113 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-33) [451c3558] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:00:54,121 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-33) [451c3558] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:00:57,129 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:00:58,111 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:00:58,111 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:00:58,111 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-1) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:00:58,113 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-1) [30db9a86] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:00:58,113 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-1) [30db9a86] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:00:58,120 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-1) [30db9a86] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:01:01,128 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:01:03,128 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:01:03,128 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:01:03,128 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-10) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:01:03,129 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-10) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:01:03,129 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-10) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:01:03,137 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-10) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:01:06,146 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:01:07,130 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:01:07,130 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:01:07,130 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-12) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:01:07,131 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-12) [7458e5a2] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:01:07,131 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-12) [7458e5a2] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:01:07,145 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-12) [7458e5a2] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:01:07,151 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-12) [7458e5a2] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:01:10,159 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:01:12,159 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:01:12,159 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:01:12,159 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-22) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:01:12,161 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-22) [1e1b6a16] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:01:12,161 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-22) [1e1b6a16] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:01:12,168 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-22) [1e1b6a16] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:01:15,175 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:01:16,161 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:01:16,161 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:01:16,161 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-30) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:01:16,163 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-30) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:01:16,163 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-30) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:01:16,169 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-30) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:01:19,177 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:01:21,177 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:01:21,177 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:01:21,177 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-64) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:01:21,178 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-64) [785bb27f] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:01:21,179 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-64) [785bb27f] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:01:21,186 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-64) [785bb27f] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:01:24,195 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:01:25,179 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:01:25,179 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:01:25,179 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-11) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:01:25,181 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-11) [4df5b8de] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:01:25,181 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-11) [4df5b8de] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:01:25,188 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-11) [4df5b8de] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:01:28,198 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:01:30,198 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:01:30,198 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:01:30,198 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-36) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:01:30,200 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-36) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:01:30,200 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-36) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:01:30,208 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-36) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:01:33,218 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:01:34,200 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:01:34,200 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:01:34,200 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-79) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:01:34,202 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-79) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:01:34,202 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-79) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:01:34,214 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-79) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:01:37,223 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:01:39,223 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:01:39,223 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:01:39,223 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-96) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:01:39,224 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-96) [721bb546] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:01:39,224 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-96) [721bb546] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:01:39,233 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-96) [721bb546] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:01:42,241 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:01:43,225 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:01:43,225 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:01:43,225 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-39) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:01:43,227 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-39) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:01:43,228 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-39) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:01:43,238 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-39) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:01:46,246 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:01:48,247 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:01:48,247 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:01:48,247 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-67) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:01:48,248 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-67) [6cea7f79] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:01:48,249 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-67) [6cea7f79] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:01:48,256 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-67) [6cea7f79] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:01:51,264 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:01:52,249 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:01:52,249 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:01:52,249 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-62) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:01:52,251 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-62) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:01:52,251 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-62) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:01:52,258 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-62) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:01:55,264 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:01:57,265 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:01:57,265 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:01:57,265 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-44) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:01:57,266 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-44) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:01:57,266 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-44) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:01:57,274 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-44) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:02:00,282 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:02:01,267 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:02:01,267 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:02:01,267 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-50) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:02:01,268 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-50) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:02:01,269 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-50) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:02:01,276 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-50) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:02:04,284 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:02:06,284 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:02:06,284 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:02:06,284 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-2) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:02:06,285 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-2) [16829b39] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:02:06,286 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-2) [16829b39] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:02:06,294 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-2) [16829b39] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:02:09,300 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:02:10,286 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:02:10,286 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:02:10,286 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-69) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:02:10,287 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-69) [294797be] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:02:10,288 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-69) [294797be] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:02:10,295 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-69) [294797be] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:02:13,304 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:02:15,304 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:02:15,304 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:02:15,304 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-16) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:02:15,305 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-16) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:02:15,305 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-16) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:02:15,313 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-16) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:02:18,321 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:02:19,306 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:02:19,306 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:02:19,306 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-37) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:02:19,308 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-37) [310dcfa0] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:02:19,308 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-37) [310dcfa0] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:02:19,316 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-37) [310dcfa0] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:02:22,324 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:02:24,324 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:02:24,324 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:02:24,324 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-84) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:02:24,325 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-84) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:02:24,325 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-84) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:02:24,333 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-84) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:02:27,340 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:02:28,326 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:02:28,326 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:02:28,326 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-10) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:02:28,328 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-10) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:02:28,328 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-10) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:02:28,339 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-10) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:02:31,348 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:02:33,348 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:02:33,348 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:02:33,348 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-45) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:02:33,349 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-45) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:02:33,349 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-45) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:02:33,356 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-45) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:02:36,366 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:02:37,350 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:02:37,350 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:02:37,350 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-48) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:02:37,352 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-48) [40606675] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:02:37,352 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-48) [40606675] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:02:37,359 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-48) [40606675] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:02:40,367 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:02:42,367 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:02:42,367 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:02:42,367 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-77) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:02:42,369 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-77) [22f3702] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:02:42,369 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-77) [22f3702] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:02:42,377 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-77) [22f3702] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:02:45,385 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:02:46,369 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:02:46,369 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:02:46,369 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-88) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:02:46,371 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-88) [f89ebd8] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:02:46,371 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-88) [f89ebd8] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:02:46,379 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-88) [f89ebd8] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:02:49,389 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:02:51,389 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:02:51,389 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:02:51,389 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-2) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:02:51,390 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-2) [16829b39] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:02:51,391 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-2) [16829b39] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:02:51,398 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-2) [16829b39] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:02:54,405 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:02:55,391 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:02:55,391 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:02:55,391 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-69) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:02:55,393 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-69) [294797be] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:02:55,393 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-69) [294797be] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:02:55,404 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-69) [294797be] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:02:58,414 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:03:00,414 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:03:00,414 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:03:00,414 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-6) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:03:00,415 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-6) [23c78bb] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:03:00,415 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-6) [23c78bb] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:03:00,425 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-6) [23c78bb] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:03:03,435 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:03:04,416 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:03:04,416 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:03:04,416 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-37) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:03:04,418 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-37) [310dcfa0] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:03:04,418 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-37) [310dcfa0] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:03:04,425 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-37) [310dcfa0] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:03:07,434 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:03:09,434 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:03:09,434 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:03:09,434 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-39) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:03:09,435 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-39) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:03:09,436 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-39) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:03:09,443 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-39) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:03:12,451 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:03:13,436 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:03:13,436 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:03:13,436 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-4) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:03:13,438 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-4) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:03:13,438 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-4) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:03:13,450 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-4) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:03:16,459 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:03:18,459 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:03:18,461 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:03:18,461 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-62) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:03:18,463 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-62) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:03:18,463 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-62) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:03:18,471 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-62) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:03:21,479 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:03:22,461 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:03:22,461 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:03:22,461 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-91) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:03:22,463 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-91) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:03:22,463 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-91) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:03:22,472 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-91) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:03:25,479 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:03:27,479 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:03:27,479 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:03:27,479 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-68) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:03:27,480 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-68) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:03:27,480 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-68) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:03:27,488 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-68) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:03:30,495 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:03:31,482 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:03:31,482 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:03:31,483 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-27) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:03:31,484 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-27) [3ede3f2d] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:03:31,484 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-27) [3ede3f2d] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:03:31,491 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-27) [3ede3f2d] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:03:34,499 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:03:36,499 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:03:36,499 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:03:36,499 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-74) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:03:36,501 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-74) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:03:36,501 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-74) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:03:36,510 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-74) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:03:39,519 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:03:40,501 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:03:40,501 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:03:40,501 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-29) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:03:40,503 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-29) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:03:40,503 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-29) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:03:40,510 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-29) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:03:43,518 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:03:45,518 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:03:45,518 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:03:45,518 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-25) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:03:45,520 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-25) [5298f326] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:03:45,520 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-25) [5298f326] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:03:45,527 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-25) [5298f326] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:03:48,535 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:03:49,520 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:03:49,520 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:03:49,520 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-1) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:03:49,522 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-1) [30db9a86] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:03:49,522 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-1) [30db9a86] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:03:49,530 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-1) [30db9a86] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:03:52,537 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:03:54,537 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:03:54,537 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:03:54,537 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-80) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:03:54,539 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-80) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:03:54,539 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-80) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:03:54,549 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-80) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:03:57,557 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:03:58,539 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:03:58,539 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:03:58,539 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-100) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:03:58,541 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-100) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:03:58,541 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-100) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:03:58,550 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-100) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:04:01,557 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:04:03,558 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:04:03,558 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:04:03,558 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-7) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:04:03,560 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-7) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:04:03,560 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-7) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:04:03,566 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-7) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:04:06,576 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:04:07,560 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:04:07,560 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:04:07,560 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-51) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:04:07,561 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-51) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:04:07,561 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-51) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:04:07,568 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-51) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:04:07,573 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-51) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:04:10,582 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:04:12,583 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:04:12,583 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:04:12,583 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-93) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:04:12,585 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-93) [4f020087] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:04:12,585 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-93) [4f020087] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:04:12,593 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-93) [4f020087] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:04:15,600 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:04:16,584 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:04:16,584 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:04:16,584 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-11) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:04:16,586 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-11) [4df5b8de] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:04:16,586 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-11) [4df5b8de] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:04:16,595 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-11) [4df5b8de] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:04:19,603 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:04:21,603 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:04:21,603 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:04:21,603 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-59) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:04:21,604 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-59) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:04:21,604 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-59) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:04:21,613 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-59) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:04:24,621 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:04:25,605 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:04:25,605 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:04:25,605 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-16) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:04:25,607 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-16) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:04:25,607 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-16) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:04:25,615 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-16) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:04:28,625 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:04:30,625 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:04:30,625 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:04:30,625 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-25) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:04:30,627 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-25) [5298f326] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:04:30,627 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-25) [5298f326] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:04:30,634 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-25) [5298f326] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:04:33,641 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:04:34,627 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:04:34,627 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:04:34,627 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-70) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:04:34,628 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-70) [562b8b0f] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:04:34,629 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-70) [562b8b0f] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:04:34,636 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-70) [562b8b0f] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:04:37,644 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:04:39,645 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:04:39,645 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:04:39,645 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-21) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:04:39,646 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-21) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:04:39,646 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-21) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:04:39,655 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-21) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:04:42,662 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:04:43,646 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:04:43,646 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:04:43,646 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-94) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:04:43,648 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-94) [5dd0314d] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:04:43,648 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-94) [5dd0314d] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:04:43,689 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-94) [5dd0314d] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:04:46,696 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:04:48,696 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:04:48,697 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:04:48,697 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-17) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:04:48,699 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-17) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:04:48,699 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-17) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:04:48,707 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-17) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:04:51,715 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:04:52,698 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:04:52,698 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:04:52,698 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-50) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:04:52,699 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-50) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:04:52,699 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-50) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:04:52,706 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-50) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:04:55,713 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:04:57,713 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:04:57,713 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:04:57,713 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-93) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:04:57,714 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-93) [4f020087] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:04:57,715 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-93) [4f020087] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:04:57,722 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-93) [4f020087] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:05:00,730 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:05:01,715 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:05:01,715 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:05:01,715 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-11) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:05:01,717 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-11) [4df5b8de] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:05:01,717 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-11) [4df5b8de] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:05:01,724 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-11) [4df5b8de] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:05:04,731 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:05:06,731 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:05:06,731 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:05:06,731 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-6) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:05:06,732 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-6) [23c78bb] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:05:06,732 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-6) [23c78bb] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:05:06,740 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-6) [23c78bb] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:05:09,748 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:05:10,733 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:05:10,733 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:05:10,733 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-25) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:05:10,735 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-25) [5298f326] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:05:10,735 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-25) [5298f326] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:05:10,742 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-25) [5298f326] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:05:13,751 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:05:15,751 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:05:15,751 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:05:15,751 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-19) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:05:15,752 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-19) [5a26db10] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:05:15,753 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-19) [5a26db10] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:05:15,759 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-19) [5a26db10] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:05:18,765 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:05:19,753 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:05:19,753 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:05:19,753 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-4) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:05:19,755 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-4) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:05:19,755 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-4) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:05:19,766 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-4) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:05:22,774 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:05:24,774 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:05:24,774 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:05:24,774 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-34) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:05:24,775 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-34) [3e42dc0b] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:05:24,775 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-34) [3e42dc0b] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:05:24,783 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-34) [3e42dc0b] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:05:27,791 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:05:28,776 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:05:28,776 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:05:28,776 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-91) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:05:28,778 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-91) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:05:28,778 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-91) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:05:28,788 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-91) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:05:31,796 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:05:33,797 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:05:33,797 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:05:33,797 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-63) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:05:33,798 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-63) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:05:33,799 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-63) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:05:33,807 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-63) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:05:36,816 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:05:37,799 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:05:37,799 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:05:37,799 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-65) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:05:37,801 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-65) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:05:37,801 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-65) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:05:37,808 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-65) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:05:40,815 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:05:42,815 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:05:42,815 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:05:42,815 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-5) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:05:42,817 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-5) [13a89c46] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:05:42,817 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-5) [13a89c46] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:05:42,824 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-5) [13a89c46] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:05:45,832 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:05:46,818 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:05:46,818 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:05:46,818 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-85) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:05:46,821 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-85) [61bce53] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:05:46,821 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-85) [61bce53] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:05:46,830 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-85) [61bce53] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:05:49,837 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:05:51,838 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:05:51,838 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:05:51,838 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-6) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:05:51,839 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-6) [23c78bb] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:05:51,839 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-6) [23c78bb] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:05:51,846 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-6) [23c78bb] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:05:54,854 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:05:55,839 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:05:55,839 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:05:55,839 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-54) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:05:55,840 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-54) [15e2b976] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:05:55,840 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-54) [15e2b976] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:05:55,848 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-54) [15e2b976] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:05:58,857 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:06:00,857 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:06:00,857 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:06:00,857 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-39) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:06:00,858 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-39) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:06:00,859 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-39) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:06:00,866 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-39) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:06:03,875 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:06:04,859 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:06:04,859 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:06:04,859 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-4) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:06:04,860 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-4) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:06:04,860 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-4) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:06:04,867 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-4) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:06:07,873 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:06:09,874 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:06:09,874 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:06:09,874 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-8) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:06:09,876 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-8) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:06:09,876 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-8) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:06:09,884 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-8) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:06:12,895 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:06:13,876 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:06:13,876 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:06:13,876 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-30) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:06:13,878 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-30) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:06:13,878 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-30) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:06:13,889 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-30) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:06:16,898 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:06:18,898 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:06:18,898 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:06:18,898 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-99) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:06:18,900 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-99) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:06:18,900 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-99) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:06:18,908 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-99) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:06:21,917 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:06:22,900 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:06:22,900 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:06:22,900 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-2) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:06:22,902 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-2) [16829b39] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:06:22,902 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-2) [16829b39] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:06:22,912 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-2) [16829b39] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:06:25,922 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:06:27,922 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:06:27,922 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:06:27,922 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-69) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:06:27,923 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-69) [294797be] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:06:27,923 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-69) [294797be] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:06:27,930 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-69) [294797be] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:06:30,938 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:06:31,924 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:06:31,924 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:06:31,924 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-85) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:06:31,925 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-85) [61bce53] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:06:31,926 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-85) [61bce53] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:06:31,932 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-85) [61bce53] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:06:34,939 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:06:36,939 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:06:36,939 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:06:36,939 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-61) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:06:36,941 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-61) [20ef7a0c] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:06:36,941 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-61) [20ef7a0c] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:06:36,948 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-61) [20ef7a0c] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:06:39,956 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:06:40,941 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:06:40,941 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:06:40,941 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-70) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:06:40,943 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-70) [562b8b0f] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:06:40,943 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-70) [562b8b0f] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:06:40,951 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-70) [562b8b0f] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:06:43,960 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:06:45,961 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:06:45,961 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:06:45,961 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-42) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:06:45,963 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-42) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:06:45,963 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-42) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:06:45,969 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-42) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:06:48,978 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:06:49,963 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:06:49,963 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:06:49,963 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-94) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:06:49,964 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-94) [5dd0314d] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:06:49,964 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-94) [5dd0314d] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:06:49,972 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-94) [5dd0314d] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:06:52,980 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:06:54,980 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:06:54,982 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:06:54,982 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-41) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:06:54,984 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-41) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:06:54,984 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-41) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:06:54,991 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-41) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:06:58,000 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:06:58,982 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:06:58,982 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:06:58,982 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-50) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:06:58,984 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-50) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:06:58,984 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-50) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:06:58,991 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-50) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:07:01,999 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:07:03,999 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:07:03,999 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:07:03,999 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-27) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:07:04,000 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-27) [3ede3f2d] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:07:04,000 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-27) [3ede3f2d] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:07:04,008 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-27) [3ede3f2d] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:07:07,017 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:07:08,002 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:07:08,002 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:07:08,002 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-5) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:07:08,004 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-5) [13a89c46] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:07:08,004 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-5) [13a89c46] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:07:08,009 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-5) [13a89c46] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:07:08,013 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-5) [13a89c46] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:07:11,020 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:07:13,021 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:07:13,021 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:07:13,021 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-58) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:07:13,022 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-58) [1a19a2ca] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:07:13,022 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-58) [1a19a2ca] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:07:13,031 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-58) [1a19a2ca] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:07:16,039 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:07:17,023 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:07:17,023 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:07:17,023 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-6) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:07:17,024 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-6) [23c78bb] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:07:17,024 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-6) [23c78bb] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:07:17,031 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-6) [23c78bb] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:07:20,040 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:07:22,040 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:07:22,040 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:07:22,040 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-61) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:07:22,042 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-61) [20ef7a0c] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:07:22,042 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-61) [20ef7a0c] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:07:22,048 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-61) [20ef7a0c] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:07:25,056 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:07:26,042 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:07:26,042 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:07:26,042 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-80) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:07:26,045 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-80) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:07:26,045 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-80) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:07:26,054 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-80) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:07:29,063 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:07:31,063 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:07:31,063 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:07:31,063 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-42) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:07:31,113 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-42) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:07:31,113 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-42) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:07:31,121 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-42) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:07:34,129 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:07:35,065 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:07:35,065 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:07:35,065 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-22) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:07:35,067 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-22) [1e1b6a16] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:07:35,067 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-22) [1e1b6a16] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:07:35,074 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-22) [1e1b6a16] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:07:38,082 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:07:40,082 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:07:40,082 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:07:40,082 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-28) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:07:40,084 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-28) [60adf93f] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:07:40,084 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-28) [60adf93f] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:07:40,092 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-28) [60adf93f] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:07:43,099 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:07:44,084 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:07:44,084 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:07:44,084 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-68) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:07:44,086 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-68) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:07:44,087 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-68) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:07:44,096 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-68) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:07:47,105 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:07:49,106 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:07:49,107 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:07:49,107 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-89) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:07:49,109 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-89) [18caed52] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:07:49,110 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-89) [18caed52] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:07:49,119 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-89) [18caed52] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:07:52,128 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:07:53,108 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:07:53,108 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:07:53,108 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-36) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:07:53,110 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-36) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:07:53,110 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-36) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:07:53,118 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-36) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:07:56,124 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:07:58,125 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:07:58,125 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:07:58,125 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-81) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:07:58,126 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-81) [2dc3b30d] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:07:58,126 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-81) [2dc3b30d] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:07:58,134 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-81) [2dc3b30d] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:08:01,142 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:08:02,127 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:08:02,127 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:08:02,127 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-49) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:08:02,129 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-49) [196d3ba4] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:08:02,129 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-49) [196d3ba4] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:08:02,137 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-49) [196d3ba4] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:08:05,145 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:08:07,146 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:08:07,146 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:08:07,146 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-70) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:08:07,147 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-70) [562b8b0f] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:08:07,147 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-70) [562b8b0f] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:08:07,155 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-70) [562b8b0f] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:08:10,162 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:08:11,148 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:08:11,148 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:08:11,148 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-14) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:08:11,150 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-14) [6343caea] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:08:11,150 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-14) [6343caea] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:08:11,158 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-14) [6343caea] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:08:14,167 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:08:16,167 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:08:16,167 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:08:16,167 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-94) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:08:16,168 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-94) [5dd0314d] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:08:16,168 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-94) [5dd0314d] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:08:16,178 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-94) [5dd0314d] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:08:19,185 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:08:20,169 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:08:20,169 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:08:20,169 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-41) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:08:20,170 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-41) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:08:20,170 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-41) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:08:20,178 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-41) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:08:23,186 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:08:25,186 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:08:25,186 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:08:25,186 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-92) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:08:25,187 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-92) [3c53e142] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:08:25,188 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-92) [3c53e142] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:08:25,196 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-92) [3c53e142] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:08:28,202 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:08:29,188 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:08:29,188 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:08:29,188 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-93) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:08:29,190 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-93) [4f020087] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:08:29,190 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-93) [4f020087] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:08:29,199 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-93) [4f020087] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:08:32,207 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:08:34,207 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:08:34,207 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:08:34,207 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-11) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:08:34,209 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-11) [4df5b8de] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:08:34,209 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-11) [4df5b8de] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:08:34,216 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-11) [4df5b8de] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:08:37,226 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:08:38,209 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:08:38,209 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:08:38,209 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-79) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:08:38,211 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-79) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:08:38,211 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-79) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:08:38,219 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-79) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:08:41,226 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:08:43,226 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:08:43,226 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:08:43,226 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-33) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:08:43,227 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-33) [451c3558] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:08:43,227 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-33) [451c3558] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:08:43,235 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-33) [451c3558] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:08:46,241 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:08:47,229 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:08:47,229 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:08:47,229 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-84) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:08:47,230 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-84) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:08:47,230 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-84) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:08:47,237 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-84) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:08:50,244 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:08:52,244 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:08:52,244 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:08:52,244 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-71) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:08:52,245 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-71) [23e814fc] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:08:52,246 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-71) [23e814fc] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:08:52,253 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-71) [23e814fc] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:08:55,261 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:08:56,246 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:08:56,246 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:08:56,246 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-12) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:08:56,248 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-12) [7458e5a2] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:08:56,248 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-12) [7458e5a2] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:08:56,256 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-12) [7458e5a2] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:08:59,264 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:09:01,264 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:09:01,264 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:09:01,264 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-7) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:09:01,265 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-7) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:09:01,265 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-7) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:09:01,273 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-7) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:09:04,282 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:09:05,266 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:09:05,266 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:09:05,266 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-30) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:09:05,268 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-30) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:09:05,269 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-30) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:09:05,278 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-30) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:09:08,286 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:09:10,286 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:09:10,286 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:09:10,286 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-9) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:09:10,287 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-9) [5c6fc8e4] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:09:10,287 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-9) [5c6fc8e4] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:09:10,295 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-9) [5c6fc8e4] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:09:13,303 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:09:14,288 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:09:14,288 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:09:14,288 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-32) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:09:14,291 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-32) [56381f88] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:09:14,291 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-32) [56381f88] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:09:14,302 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-32) [56381f88] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:09:17,310 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:09:19,311 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:09:19,311 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:09:19,311 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-75) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:09:19,313 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-75) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:09:19,313 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-75) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:09:19,321 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-75) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:09:22,329 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:09:23,313 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:09:23,313 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:09:23,313 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-16) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:09:23,315 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-16) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:09:23,315 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-16) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:09:23,323 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-16) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:09:26,332 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:09:28,332 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:09:28,332 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:09:28,332 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-25) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:09:28,333 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-25) [5298f326] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:09:28,333 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-25) [5298f326] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:09:28,343 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-25) [5298f326] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:09:31,350 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:09:32,334 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:09:32,334 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:09:32,334 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-84) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:09:32,336 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-84) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:09:32,336 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-84) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:09:32,346 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-84) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:09:35,354 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:09:37,354 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:09:37,354 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:09:37,354 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-15) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:09:37,356 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-15) [3a4dd3fc] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:09:37,356 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-15) [3a4dd3fc] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:09:37,363 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-15) [3a4dd3fc] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:09:40,371 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:09:41,356 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:09:41,356 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:09:41,356 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-34) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:09:41,358 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-34) [3e42dc0b] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:09:41,358 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-34) [3e42dc0b] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:09:41,366 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-34) [3e42dc0b] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:09:44,375 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:09:46,375 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:09:46,375 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:09:46,375 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-44) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:09:46,376 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-44) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:09:46,377 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-44) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:09:46,384 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-44) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:09:49,392 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:09:50,377 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:09:50,377 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:09:50,377 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-63) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:09:50,379 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-63) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:09:50,379 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-63) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:09:50,387 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-63) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:09:53,394 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:09:55,394 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:09:55,394 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:09:55,394 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-47) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:09:55,396 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-47) [4028fe5f] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:09:55,396 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-47) [4028fe5f] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:09:55,404 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-47) [4028fe5f] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:09:58,412 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:09:59,396 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:09:59,396 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:09:59,396 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-18) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:09:59,399 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-18) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:09:59,399 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-18) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:09:59,409 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-18) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:10:02,418 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:10:04,418 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:10:04,418 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:10:04,418 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-29) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:10:04,419 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-29) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:10:04,420 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-29) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:10:04,428 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-29) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:10:07,435 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:10:08,420 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:10:08,420 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:10:08,420 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-55) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:10:08,422 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-55) [7a4dea71] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:10:08,422 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-55) [7a4dea71] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:10:08,432 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-55) [7a4dea71] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:10:08,439 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-55) [7a4dea71] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:10:11,447 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:10:13,448 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:10:13,448 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:10:13,448 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-53) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:10:13,449 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-53) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:10:13,449 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-53) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:10:13,457 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-53) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:10:16,464 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:10:17,450 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:10:17,450 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:10:17,450 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-14) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:10:17,451 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-14) [6343caea] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:10:17,452 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-14) [6343caea] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:10:17,459 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-14) [6343caea] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:10:20,467 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:10:22,468 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:10:22,468 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:10:22,468 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-38) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:10:22,470 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-38) [56c37b5] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:10:22,470 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-38) [56c37b5] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:10:22,476 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-38) [56c37b5] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:10:25,483 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:10:26,470 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:10:26,470 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:10:26,470 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-17) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:10:26,471 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-17) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:10:26,471 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-17) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:10:26,478 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-17) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:10:29,488 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:10:31,488 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:10:31,488 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:10:31,488 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-30) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:10:31,489 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-30) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:10:31,490 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-30) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:10:31,497 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-30) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:10:34,503 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:10:35,490 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:10:35,490 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:10:35,490 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-65) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:10:35,491 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-65) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:10:35,492 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-65) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:10:35,500 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-65) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:10:38,506 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:10:40,507 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:10:40,507 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:10:40,507 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-74) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:10:40,508 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-74) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:10:40,508 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-74) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:10:40,517 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-74) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:10:43,526 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:10:44,508 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:10:44,508 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:10:44,508 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-75) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:10:44,510 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-75) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:10:44,510 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-75) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:10:44,520 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-75) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:10:47,528 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:10:49,528 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:10:49,529 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:10:49,529 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-66) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:10:49,531 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-66) [e582aa6] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:10:49,531 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-66) [e582aa6] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:10:49,538 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-66) [e582aa6] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:10:52,546 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:10:53,530 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:10:53,530 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:10:53,530 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-55) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:10:53,531 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-55) [7a4dea71] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:10:53,531 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-55) [7a4dea71] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:10:53,540 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-55) [7a4dea71] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:10:56,547 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:10:58,548 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:10:58,548 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:10:58,548 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-53) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:10:58,549 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-53) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:10:58,549 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-53) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:10:58,558 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-53) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:11:01,566 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:11:02,550 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:11:02,550 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:11:02,550 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-4) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:11:02,551 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-4) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:11:02,552 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-4) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:11:02,560 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-4) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:11:05,568 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:11:07,568 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:11:07,568 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:11:07,569 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-22) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:11:07,571 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-22) [1e1b6a16] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:11:07,571 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-22) [1e1b6a16] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:11:07,577 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-22) [1e1b6a16] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:11:10,586 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:11:11,570 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:11:11,570 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:11:11,570 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-50) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:11:11,572 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-50) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:11:11,572 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-50) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:11:11,579 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-50) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:11:14,588 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:11:16,589 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:11:16,589 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:11:16,589 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-99) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:11:16,590 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-99) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:11:16,591 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-99) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:11:16,598 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-99) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:11:19,605 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:11:20,591 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:11:20,591 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:11:20,591 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-89) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:11:20,593 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-89) [18caed52] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:11:20,593 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-89) [18caed52] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:11:20,600 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-89) [18caed52] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:11:23,607 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:11:25,608 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:11:25,608 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:11:25,608 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-36) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:11:25,610 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-36) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:11:25,610 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-36) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:11:25,617 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-36) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:11:28,624 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:11:29,609 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:11:29,609 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:11:29,609 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-81) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:11:29,611 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-81) [2dc3b30d] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:11:29,611 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-81) [2dc3b30d] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:11:29,621 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-81) [2dc3b30d] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:11:32,631 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:11:34,631 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:11:34,631 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:11:34,631 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-40) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:11:34,632 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-40) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:11:34,632 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-40) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:11:34,640 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-40) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:11:37,658 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:11:38,633 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:11:38,633 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:11:38,633 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-10) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:11:38,635 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-10) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:11:38,635 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-10) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:11:38,646 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-10) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:11:41,655 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:11:43,655 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:11:43,655 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:11:43,655 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-43) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:11:43,657 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-43) [2d3b2827] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:11:43,657 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-43) [2d3b2827] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:11:43,667 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-43) [2d3b2827] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:11:46,676 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:11:47,657 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:11:47,657 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:11:47,657 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-48) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:11:47,659 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-48) [40606675] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:11:47,659 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-48) [40606675] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:11:47,669 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-48) [40606675] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:11:50,679 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:11:52,680 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:11:52,680 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:11:52,680 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-91) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:11:52,681 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-91) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:11:52,681 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-91) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:11:52,689 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-91) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:11:55,695 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:11:56,681 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:11:56,681 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:11:56,681 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-63) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:11:56,683 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-63) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:11:56,683 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-63) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:11:56,690 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-63) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:11:59,697 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:12:01,697 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:12:01,697 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:12:01,697 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-9) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:12:01,699 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-9) [5c6fc8e4] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:12:01,699 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-9) [5c6fc8e4] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:12:01,709 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-9) [5c6fc8e4] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:12:04,720 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:12:05,699 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:12:05,699 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:12:05,699 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-11) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:12:05,702 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-11) [4df5b8de] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:12:05,702 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-11) [4df5b8de] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:12:05,711 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-11) [4df5b8de] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:12:08,719 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:12:10,720 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:12:10,720 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:12:10,720 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-75) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:12:10,722 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-75) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:12:10,722 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-75) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:12:10,729 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-75) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:12:13,736 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:12:14,722 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:12:14,722 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:12:14,722 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-33) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:12:14,723 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-33) [451c3558] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:12:14,723 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-33) [451c3558] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:12:14,731 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-33) [451c3558] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:12:17,738 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:12:19,738 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:12:19,738 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:12:19,740 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-83) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:12:19,743 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-83) [6d722a1d] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:12:19,743 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-83) [6d722a1d] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:12:19,760 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-83) [6d722a1d] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:12:22,769 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:12:23,740 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:12:23,740 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:12:23,740 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-71) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:12:23,742 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-71) [23e814fc] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:12:23,742 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-71) [23e814fc] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:12:23,750 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-71) [23e814fc] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:12:26,756 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:12:28,757 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:12:28,757 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:12:28,757 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-12) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:12:28,759 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-12) [7458e5a2] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:12:28,759 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-12) [7458e5a2] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:12:28,767 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-12) [7458e5a2] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:12:31,776 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:12:32,759 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:12:32,759 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:12:32,759 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-48) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:12:32,760 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-48) [40606675] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:12:32,760 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-48) [40606675] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:12:32,768 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-48) [40606675] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:12:35,777 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:12:37,778 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:12:37,779 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:12:37,780 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-28) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:12:37,781 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-28) [60adf93f] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:12:37,781 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-28) [60adf93f] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:12:37,790 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-28) [60adf93f] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:12:40,797 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:12:41,780 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:12:41,780 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:12:41,780 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-95) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:12:41,782 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-95) [4bd2250] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:12:41,782 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-95) [4bd2250] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:12:41,790 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-95) [4bd2250] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:12:44,799 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:12:46,799 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:12:46,799 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:12:46,799 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-23) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:12:46,800 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-23) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:12:46,800 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-23) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:12:46,807 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-23) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:12:49,815 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:12:50,801 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:12:50,801 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:12:50,801 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-69) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:12:50,803 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-69) [294797be] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:12:50,803 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-69) [294797be] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:12:50,810 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-69) [294797be] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:12:53,817 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:12:55,818 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:12:55,818 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:12:55,818 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-57) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:12:55,820 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-57) [16b48795] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:12:55,820 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-57) [16b48795] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:12:55,827 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-57) [16b48795] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:12:58,835 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:12:59,820 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:12:59,820 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:12:59,820 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-33) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:12:59,822 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-33) [451c3558] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:12:59,822 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-33) [451c3558] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:12:59,832 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-33) [451c3558] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:13:02,841 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:13:04,841 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:13:04,841 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:13:04,841 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-84) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:13:04,842 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-84) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:13:04,842 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-84) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:13:04,850 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-84) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:13:07,859 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:13:08,843 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:13:08,843 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:13:08,843 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-43) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:13:08,844 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-43) [2d3b2827] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:13:08,844 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-43) [2d3b2827] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:13:08,851 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-43) [2d3b2827] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:13:08,856 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-43) [2d3b2827] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:13:11,863 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:13:13,864 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:13:13,864 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:13:13,864 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-34) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:13:13,866 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-34) [3e42dc0b] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:13:13,866 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-34) [3e42dc0b] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:13:13,874 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-34) [3e42dc0b] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:13:16,882 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:13:17,866 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:13:17,866 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:13:17,866 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-91) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:13:17,868 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-91) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:13:17,868 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-91) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:13:17,875 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-91) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:13:20,884 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:13:22,884 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:13:22,884 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:13:22,884 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-28) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:13:22,885 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-28) [60adf93f] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:13:22,886 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-28) [60adf93f] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:13:22,893 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-28) [60adf93f] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:13:25,902 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:13:26,886 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:13:26,886 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:13:26,886 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-47) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:13:26,887 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-47) [4028fe5f] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:13:26,887 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-47) [4028fe5f] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:13:26,895 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-47) [4028fe5f] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:13:29,902 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:13:31,902 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:13:31,902 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:13:31,902 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-5) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:13:31,903 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-5) [13a89c46] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:13:31,903 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-5) [13a89c46] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:13:31,912 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-5) [13a89c46] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:13:34,920 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:13:35,904 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:13:35,904 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:13:35,904 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-85) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:13:35,906 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-85) [61bce53] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:13:35,906 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-85) [61bce53] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:13:35,913 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-85) [61bce53] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:13:38,921 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:13:40,921 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:13:40,921 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:13:40,921 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-37) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:13:40,922 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-37) [310dcfa0] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:13:40,922 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-37) [310dcfa0] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:13:40,930 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-37) [310dcfa0] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:13:43,938 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:13:44,923 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:13:44,923 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:13:44,923 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-55) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:13:44,925 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-55) [7a4dea71] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:13:44,925 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-55) [7a4dea71] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:13:44,932 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-55) [7a4dea71] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:13:47,940 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:13:49,940 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:13:49,940 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:13:49,940 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-53) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:13:49,941 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-53) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:13:49,941 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-53) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:13:49,949 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-53) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:13:52,958 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:13:53,942 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:13:53,942 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:13:53,942 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-43) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:13:53,944 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-43) [2d3b2827] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:13:53,944 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-43) [2d3b2827] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:13:53,951 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-43) [2d3b2827] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:13:56,958 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:13:58,958 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:13:58,958 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:13:58,958 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-34) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:13:58,959 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-34) [3e42dc0b] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:13:58,959 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-34) [3e42dc0b] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:13:58,967 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-34) [3e42dc0b] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:14:01,975 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:14:02,960 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:14:02,960 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:14:02,960 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-44) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:14:02,962 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-44) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:14:02,962 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-44) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:14:02,969 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-44) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:14:05,976 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:14:07,977 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:14:07,978 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:14:07,978 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-95) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:14:07,980 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-95) [4bd2250] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:14:07,980 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-95) [4bd2250] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:14:07,988 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-95) [4bd2250] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:14:10,998 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:14:11,979 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:14:11,979 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:14:11,979 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-89) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:14:11,981 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-89) [18caed52] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:14:11,981 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-89) [18caed52] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:14:11,990 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-89) [18caed52] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:14:15,000 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:14:17,000 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:14:17,000 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:14:17,000 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-69) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:14:17,001 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-69) [294797be] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:14:17,001 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-69) [294797be] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:14:17,009 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-69) [294797be] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:14:20,017 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:14:21,002 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:14:21,002 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:14:21,002 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-81) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:14:21,004 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-81) [2dc3b30d] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:14:21,004 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-81) [2dc3b30d] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:14:21,011 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-81) [2dc3b30d] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:14:24,018 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:14:26,018 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:14:26,018 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:14:26,018 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-37) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:14:26,019 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-37) [310dcfa0] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:14:26,019 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-37) [310dcfa0] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:14:26,027 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-37) [310dcfa0] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:14:29,035 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:14:30,020 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:14:30,020 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:14:30,020 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-78) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:14:30,023 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-78) [4e91cc97] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:14:30,023 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-78) [4e91cc97] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:14:30,031 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-78) [4e91cc97] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:14:33,038 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:14:35,039 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:14:35,039 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:14:35,039 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-45) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:14:35,040 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-45) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:14:35,040 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-45) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:14:35,048 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-45) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:14:38,058 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:14:39,040 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:14:39,040 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:14:39,040 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-94) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:14:39,041 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-94) [5dd0314d] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:14:39,041 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-94) [5dd0314d] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:14:39,049 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-94) [5dd0314d] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:14:42,057 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:14:44,057 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:14:44,057 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:14:44,057 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-77) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:14:44,058 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-77) [22f3702] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:14:44,059 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-77) [22f3702] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:14:44,066 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-77) [22f3702] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:14:47,075 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:14:48,059 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:14:48,059 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:14:48,059 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-92) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:14:48,061 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-92) [3c53e142] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:14:48,061 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-92) [3c53e142] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:14:48,068 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-92) [3c53e142] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:14:51,076 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:14:53,076 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:14:53,076 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:14:53,076 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-93) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:14:53,077 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-93) [4f020087] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:14:53,078 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-93) [4f020087] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:14:53,085 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-93) [4f020087] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:14:56,092 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:14:57,078 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:14:57,078 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:14:57,078 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-74) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:14:57,080 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-74) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:14:57,080 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-74) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:14:57,088 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-74) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:15:00,095 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:15:02,095 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:15:02,095 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:15:02,095 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-79) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:15:02,096 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-79) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:15:02,096 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-79) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:15:02,104 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-79) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:15:05,114 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:15:06,097 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:15:06,097 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:15:06,097 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-54) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:15:06,098 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-54) [15e2b976] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:15:06,098 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-54) [15e2b976] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:15:06,105 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-54) [15e2b976] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:15:09,114 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:15:11,114 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:15:11,114 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:15:11,114 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-39) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:15:11,115 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-39) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:15:11,115 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-39) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:15:11,123 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-39) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:15:14,131 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:15:15,116 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:15:15,116 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:15:15,116 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-42) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:15:15,119 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-42) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:15:15,119 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-42) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:15:15,129 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-42) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:15:18,138 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:15:20,138 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:15:20,138 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:15:20,138 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-38) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:15:20,139 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-38) [56c37b5] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:15:20,140 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-38) [56c37b5] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:15:20,147 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-38) [56c37b5] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:15:23,155 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:15:24,140 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:15:24,140 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:15:24,140 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-22) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:15:24,142 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-22) [1e1b6a16] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:15:24,142 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-22) [1e1b6a16] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:15:24,149 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-22) [1e1b6a16] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:15:27,155 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:15:29,155 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:15:29,155 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:15:29,155 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-63) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:15:29,156 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-63) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:15:29,157 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-63) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:15:29,163 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-63) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:15:32,169 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:15:33,157 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:15:33,157 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:15:33,157 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-95) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:15:33,159 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-95) [4bd2250] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:15:33,159 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-95) [4bd2250] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:15:33,178 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-95) [4bd2250] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:15:36,186 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:15:38,186 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:15:38,186 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:15:38,186 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-23) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:15:38,187 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-23) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:15:38,187 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-23) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:15:38,196 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-23) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:15:41,204 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:15:42,188 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:15:42,188 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:15:42,188 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-69) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:15:42,189 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-69) [294797be] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:15:42,190 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-69) [294797be] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:15:42,198 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-69) [294797be] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:15:45,213 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:15:47,213 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:15:47,213 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:15:47,213 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-66) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:15:47,214 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-66) [e582aa6] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:15:47,214 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-66) [e582aa6] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:15:47,222 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-66) [e582aa6] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:15:50,231 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:15:51,215 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:15:51,215 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:15:51,215 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-61) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:15:51,217 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-61) [20ef7a0c] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:15:51,217 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-61) [20ef7a0c] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:15:51,225 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-61) [20ef7a0c] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:15:54,231 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:15:56,231 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:15:56,231 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:15:56,233 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-80) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:15:56,235 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-80) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:15:56,235 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-80) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:15:56,244 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-80) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:15:59,253 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:16:00,233 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:16:00,233 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:16:00,233 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-43) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:16:00,235 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-43) [2d3b2827] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:16:00,235 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-43) [2d3b2827] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:16:00,245 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-43) [2d3b2827] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:16:03,253 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:16:05,253 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:16:05,253 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:16:05,253 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-8) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:16:05,254 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-8) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:16:05,254 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-8) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:16:05,262 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-8) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:16:08,271 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:16:09,255 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:16:09,255 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:16:09,255 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-97) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:16:09,257 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-97) [54081188] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:16:09,257 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-97) [54081188] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:16:09,272 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-97) [54081188] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:16:09,279 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-97) [54081188] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:16:12,288 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:16:14,288 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:16:14,288 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:16:14,288 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-27) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:16:14,290 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-27) [3ede3f2d] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:16:14,290 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-27) [3ede3f2d] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:16:14,300 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-27) [3ede3f2d] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:16:17,309 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:16:18,290 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:16:18,290 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:16:18,290 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-89) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:16:18,292 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-89) [18caed52] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:16:18,292 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-89) [18caed52] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:16:18,300 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-89) [18caed52] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:16:21,307 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:16:23,307 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:16:23,307 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:16:23,307 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-23) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:16:23,308 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-23) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:16:23,308 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-23) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:16:23,316 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-23) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:16:26,325 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:16:27,309 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:16:27,309 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:16:27,309 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-29) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:16:27,311 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-29) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:16:27,311 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-29) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:16:27,325 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-29) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:16:30,331 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:16:32,331 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:16:32,331 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:16:32,331 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-66) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:16:32,334 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-66) [e582aa6] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:16:32,334 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-66) [e582aa6] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:16:32,341 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-66) [e582aa6] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:16:35,349 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:16:36,334 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:16:36,334 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:16:36,334 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-78) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:16:36,336 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-78) [4e91cc97] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:16:36,336 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-78) [4e91cc97] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:16:36,343 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-78) [4e91cc97] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:16:39,349 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:16:41,349 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:16:41,349 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:16:41,349 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-15) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:16:41,351 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-15) [3a4dd3fc] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:16:41,351 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-15) [3a4dd3fc] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:16:41,359 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-15) [3a4dd3fc] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:16:44,367 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:16:45,351 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:16:45,351 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:16:45,351 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-20) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:16:45,353 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-20) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:16:45,353 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-20) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:16:45,360 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-20) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:16:48,367 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:16:50,368 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:16:50,368 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:16:50,368 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-76) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:16:50,370 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-76) [1bdd13e4] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:16:50,370 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-76) [1bdd13e4] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:16:50,378 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-76) [1bdd13e4] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:16:53,386 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:16:54,370 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:16:54,370 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:16:54,370 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-88) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:16:54,371 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-88) [f89ebd8] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:16:54,372 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-88) [f89ebd8] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:16:54,379 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-88) [f89ebd8] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:16:57,387 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:16:59,387 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:16:59,387 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:16:59,387 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-31) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:16:59,388 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-31) [2eda49c0] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:16:59,388 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-31) [2eda49c0] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:16:59,396 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-31) [2eda49c0] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:17:02,404 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:17:03,389 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:17:03,389 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:17:03,389 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-5) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:17:03,390 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-5) [13a89c46] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:17:03,391 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-5) [13a89c46] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:17:03,399 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-5) [13a89c46] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:17:06,406 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:17:08,406 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:17:08,406 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:17:08,408 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-16) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:17:08,409 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-16) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:17:08,409 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-16) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:17:08,416 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-16) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:17:11,422 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:17:12,408 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:17:12,408 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:17:12,408 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-1) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:17:12,410 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-1) [30db9a86] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:17:12,410 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-1) [30db9a86] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:17:12,420 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-1) [30db9a86] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:17:15,431 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:17:17,431 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:17:17,431 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:17:17,431 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-19) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:17:17,433 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-19) [5a26db10] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:17:17,433 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-19) [5a26db10] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:17:17,440 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-19) [5a26db10] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:17:20,449 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:17:21,433 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:17:21,433 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:17:21,433 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-21) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:17:21,435 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-21) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:17:21,435 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-21) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:17:21,443 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-21) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:17:24,452 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:17:26,452 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:17:26,452 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:17:26,452 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-45) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:17:26,453 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-45) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:17:26,453 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-45) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:17:26,463 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-45) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:17:29,470 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:17:30,454 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:17:30,454 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:17:30,454 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-34) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:17:30,456 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-34) [3e42dc0b] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:17:30,456 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-34) [3e42dc0b] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:17:30,464 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-34) [3e42dc0b] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:17:33,472 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:17:35,472 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:17:35,472 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:17:35,472 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-77) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:17:35,474 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-77) [22f3702] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:17:35,474 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-77) [22f3702] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:17:35,481 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-77) [22f3702] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:17:38,491 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:17:39,474 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:17:39,474 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:17:39,474 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-99) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:17:39,476 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-99) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:17:39,476 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-99) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:17:39,483 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-99) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:17:42,489 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:17:44,489 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:17:44,489 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:17:44,489 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-2) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:17:44,490 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-2) [16829b39] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:17:44,490 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-2) [16829b39] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:17:44,498 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-2) [16829b39] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:17:47,504 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:17:48,491 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:17:48,491 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:17:48,491 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-23) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:17:48,492 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-23) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:17:48,492 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-23) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:17:48,502 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-23) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:17:51,512 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:17:53,512 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:17:53,512 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:17:53,512 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-79) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:17:53,513 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-79) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:17:53,514 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-79) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:17:53,520 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-79) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:17:56,528 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:17:57,514 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:17:57,514 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:17:57,514 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-40) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:17:57,516 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-40) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:17:57,516 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-40) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:17:57,524 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-40) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:18:00,531 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:18:02,532 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:18:02,532 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:18:02,532 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-83) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:18:02,533 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-83) [6d722a1d] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:18:02,533 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-83) [6d722a1d] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:18:02,540 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-83) [6d722a1d] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:18:05,549 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:18:06,534 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:18:06,534 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:18:06,534 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-42) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:18:06,536 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-42) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:18:06,536 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-42) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:18:06,544 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-42) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:18:09,552 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:18:11,552 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:18:11,552 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:18:11,552 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-94) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:18:11,553 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-94) [5dd0314d] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:18:11,553 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-94) [5dd0314d] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:18:11,561 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-94) [5dd0314d] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:18:14,569 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:18:15,554 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:18:15,554 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:18:15,554 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-51) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:18:15,556 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-51) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:18:15,556 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-51) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:18:15,569 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-51) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:18:18,579 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:18:20,579 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:18:20,579 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:18:20,579 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-88) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:18:20,580 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-88) [f89ebd8] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:18:20,581 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-88) [f89ebd8] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:18:20,587 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-88) [f89ebd8] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:18:23,595 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:18:24,581 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:18:24,581 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:18:24,581 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-95) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:18:24,582 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-95) [4bd2250] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:18:24,583 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-95) [4bd2250] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:18:24,590 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-95) [4bd2250] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:18:27,598 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:18:29,598 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:18:29,598 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:18:29,598 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-5) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:18:29,599 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-5) [13a89c46] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:18:29,599 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-5) [13a89c46] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:18:29,607 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-5) [13a89c46] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:18:32,613 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:18:33,600 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:18:33,600 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:18:33,600 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-75) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:18:33,601 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-75) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:18:33,601 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-75) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:18:33,608 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-75) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:18:36,616 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:18:38,616 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:18:38,617 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:18:38,618 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-96) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:18:38,619 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-96) [721bb546] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:18:38,619 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-96) [721bb546] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:18:38,626 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-96) [721bb546] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:18:41,633 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:18:42,618 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:18:42,618 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:18:42,618 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-61) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:18:42,620 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-61) [20ef7a0c] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:18:42,620 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-61) [20ef7a0c] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:18:42,630 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-61) [20ef7a0c] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:18:45,640 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:18:47,640 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:18:47,640 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:18:47,640 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-71) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:18:47,641 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-71) [23e814fc] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:18:47,641 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-71) [23e814fc] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:18:47,649 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-71) [23e814fc] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:18:50,656 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:18:51,642 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:18:51,642 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:18:51,642 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-26) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:18:51,644 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-26) [13b06b4d] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:18:51,644 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-26) [13b06b4d] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:18:51,651 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-26) [13b06b4d] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:18:54,659 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:18:56,659 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:18:56,659 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:18:56,659 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-86) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:18:56,661 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-86) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:18:56,662 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-86) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:18:56,670 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-86) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:18:59,679 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:19:00,661 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:19:00,661 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:19:00,661 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-13) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:19:00,663 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-13) [7246829] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:19:00,663 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-13) [7246829] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:19:00,671 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-13) [7246829] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:19:03,679 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:19:05,679 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:19:05,679 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:19:05,679 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-68) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:19:05,680 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-68) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:19:05,680 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-68) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:19:05,687 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-68) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:19:08,694 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:19:09,681 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:19:09,681 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:19:09,681 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-89) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:19:09,682 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-89) [18caed52] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:19:09,682 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-89) [18caed52] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:19:09,690 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-89) [18caed52] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:19:09,696 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-89) [18caed52] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:19:12,704 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:19:14,705 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:19:14,705 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:19:14,705 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-58) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:19:14,706 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-58) [1a19a2ca] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:19:14,707 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-58) [1a19a2ca] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:19:14,715 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-58) [1a19a2ca] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:19:17,722 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:19:18,707 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:19:18,707 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:19:18,707 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-6) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:19:18,708 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-6) [23c78bb] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:19:18,708 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-6) [23c78bb] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:19:18,717 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-6) [23c78bb] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:19:21,725 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:19:23,726 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:19:23,726 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:19:23,726 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-96) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:19:23,727 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-96) [721bb546] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:19:23,727 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-96) [721bb546] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:19:23,736 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-96) [721bb546] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:19:26,744 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:19:27,728 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:19:27,728 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:19:27,728 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-84) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:19:27,729 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-84) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:19:27,729 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-84) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:19:27,737 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-84) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:19:30,753 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:19:32,753 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:19:32,753 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:19:32,753 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-14) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:19:32,754 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-14) [6343caea] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:19:32,754 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-14) [6343caea] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:19:32,761 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-14) [6343caea] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:19:35,767 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:19:36,755 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:19:36,755 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:19:36,756 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-7) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:19:36,757 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-7) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:19:36,757 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-7) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:19:36,764 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-7) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:19:39,770 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:19:41,771 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:19:41,771 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:19:41,771 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-91) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:19:41,773 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-91) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:19:41,773 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-91) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:19:41,782 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-91) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:19:44,790 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:19:45,773 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:19:45,773 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:19:45,773 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-63) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:19:45,775 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-63) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:19:45,775 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-63) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:19:45,785 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-63) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:19:48,795 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:19:50,795 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:19:50,795 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:19:50,795 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-68) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:19:50,798 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-68) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:19:50,798 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-68) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:19:50,810 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-68) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:19:53,820 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:19:54,797 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:19:54,797 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:19:54,797 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-2) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:19:54,800 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-2) [16829b39] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:19:54,800 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-2) [16829b39] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:19:54,809 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-2) [16829b39] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:19:57,819 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:19:59,819 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:19:59,819 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:19:59,819 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-69) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:19:59,821 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-69) [294797be] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:19:59,821 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-69) [294797be] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:19:59,829 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-69) [294797be] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:20:02,836 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:20:03,821 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:20:03,821 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:20:03,821 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-54) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:20:03,823 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-54) [15e2b976] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:20:03,823 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-54) [15e2b976] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:20:03,835 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-54) [15e2b976] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:20:06,843 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:20:08,843 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:20:08,843 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:20:08,843 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-19) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:20:08,844 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-19) [5a26db10] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:20:08,844 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-19) [5a26db10] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:20:08,852 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-19) [5a26db10] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:20:11,860 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:20:12,845 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:20:12,845 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:20:12,845 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-15) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:20:12,847 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-15) [3a4dd3fc] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:20:12,847 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-15) [3a4dd3fc] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:20:12,856 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-15) [3a4dd3fc] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:20:15,865 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:20:17,866 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:20:17,866 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:20:17,866 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-20) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:20:17,867 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-20) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:20:17,867 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-20) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:20:17,875 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-20) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:20:20,882 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:20:21,868 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:20:21,868 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:20:21,868 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-76) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:20:21,870 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-76) [1bdd13e4] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:20:21,870 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-76) [1bdd13e4] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:20:21,876 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-76) [1bdd13e4] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:20:24,883 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:20:26,883 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:20:26,883 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:20:26,883 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-28) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:20:26,885 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-28) [60adf93f] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:20:26,885 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-28) [60adf93f] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:20:26,891 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-28) [60adf93f] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:20:29,900 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:20:30,885 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:20:30,885 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:20:30,885 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-31) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:20:30,887 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-31) [2eda49c0] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:20:30,887 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-31) [2eda49c0] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:20:30,896 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-31) [2eda49c0] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:20:33,906 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:20:35,906 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:20:35,906 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:20:35,906 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-89) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:20:35,908 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-89) [18caed52] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:20:35,908 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-89) [18caed52] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:20:35,918 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-89) [18caed52] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:20:38,941 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:20:39,908 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:20:39,908 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:20:39,908 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-75) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:20:39,909 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-75) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:20:39,909 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-75) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:20:39,917 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-75) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:20:42,925 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:20:44,925 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:20:44,925 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:20:44,925 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-6) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:20:44,926 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-6) [23c78bb] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:20:44,926 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-6) [23c78bb] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:20:44,935 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-6) [23c78bb] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:20:47,941 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:20:48,927 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:20:48,927 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:20:48,927 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-70) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:20:48,928 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-70) [562b8b0f] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:20:48,928 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-70) [562b8b0f] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:20:48,935 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-70) [562b8b0f] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:20:51,943 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:20:53,943 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:20:53,943 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:20:53,943 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-67) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:20:53,944 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-67) [6cea7f79] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:20:53,944 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-67) [6cea7f79] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:20:53,951 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-67) [6cea7f79] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:20:56,959 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:20:57,945 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:20:57,945 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:20:57,945 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-82) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:20:57,946 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-82) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:20:57,946 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-82) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:20:57,955 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-82) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:21:00,964 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:21:02,964 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:21:02,964 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:21:02,964 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-7) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:21:02,967 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-7) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:21:02,967 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-7) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:21:02,975 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-7) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:21:05,983 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:21:06,966 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:21:06,966 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:21:06,966 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-77) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:21:06,968 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-77) [22f3702] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:21:06,968 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-77) [22f3702] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:21:06,975 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-77) [22f3702] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:21:09,981 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:21:11,982 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:21:11,982 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:21:11,982 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-9) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:21:11,984 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-9) [5c6fc8e4] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:21:11,984 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-9) [5c6fc8e4] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:21:11,992 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-9) [5c6fc8e4] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:21:14,999 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:21:15,984 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:21:15,984 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:21:15,984 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-11) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:21:15,985 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-11) [4df5b8de] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:21:15,985 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-11) [4df5b8de] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:21:15,993 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-11) [4df5b8de] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:21:18,999 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:21:21,000 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:21:21,000 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:21:21,000 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-23) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:21:21,001 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-23) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:21:21,001 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-23) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:21:21,010 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-23) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:21:24,019 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:21:25,001 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:21:25,001 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:21:25,001 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-57) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:21:25,002 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-57) [16b48795] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:21:25,003 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-57) [16b48795] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:21:25,010 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-57) [16b48795] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:21:28,017 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:21:30,017 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:21:30,017 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:21:30,017 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-40) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:21:30,018 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-40) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:21:30,018 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-40) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:21:30,027 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-40) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:21:33,034 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:21:34,019 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:21:34,019 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:21:34,019 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-83) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:21:34,021 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-83) [6d722a1d] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:21:34,021 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-83) [6d722a1d] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:21:34,027 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-83) [6d722a1d] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:21:37,034 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:21:39,034 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:21:39,035 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:21:39,036 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-42) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:21:39,037 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-42) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:21:39,037 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-42) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:21:39,044 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-42) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:21:42,052 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:21:43,036 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:21:43,036 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:21:43,036 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-38) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:21:43,038 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-38) [56c37b5] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:21:43,038 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-38) [56c37b5] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:21:43,045 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-38) [56c37b5] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:21:46,053 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:21:48,053 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:21:48,053 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:21:48,053 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-76) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:21:48,055 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-76) [1bdd13e4] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:21:48,055 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-76) [1bdd13e4] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:21:48,062 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-76) [1bdd13e4] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:21:51,069 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:21:52,055 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:21:52,055 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:21:52,055 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-88) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:21:52,057 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-88) [f89ebd8] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:21:52,057 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-88) [f89ebd8] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:21:52,064 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-88) [f89ebd8] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:21:55,070 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:21:57,071 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:21:57,071 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:21:57,071 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-31) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:21:57,072 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-31) [2eda49c0] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:21:57,072 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-31) [2eda49c0] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:21:57,080 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-31) [2eda49c0] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:22:00,087 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:22:01,073 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:22:01,073 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:22:01,073 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-59) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:22:01,074 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-59) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:22:01,074 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-59) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:22:01,082 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-59) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:22:04,098 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:22:06,098 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:22:06,098 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:22:06,098 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-23) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:22:06,100 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-23) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:22:06,100 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-23) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:22:06,109 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-23) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:22:09,117 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:22:10,100 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:22:10,100 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:22:10,100 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-60) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:22:10,102 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-60) [7bc6f040] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:22:10,102 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-60) [7bc6f040] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:22:10,111 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-60) [7bc6f040] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:22:10,117 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-60) [7bc6f040] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:22:13,125 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:22:15,126 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:22:15,126 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:22:15,126 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-61) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:22:15,127 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-61) [20ef7a0c] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:22:15,128 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-61) [20ef7a0c] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:22:15,135 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-61) [20ef7a0c] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:22:18,143 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:22:19,128 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:22:19,128 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:22:19,128 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-71) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:22:19,129 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-71) [23e814fc] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:22:19,129 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-71) [23e814fc] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:22:19,137 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-71) [23e814fc] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:22:22,146 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:22:24,146 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:22:24,146 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:22:24,146 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-42) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:22:24,147 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-42) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:22:24,147 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-42) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:22:24,155 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-42) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:22:27,163 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:22:28,149 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:22:28,149 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:22:28,149 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-86) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:22:28,152 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-86) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:22:28,152 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-86) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:22:28,163 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-86) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:22:31,172 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:22:33,172 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:22:33,172 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:22:33,172 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-13) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:22:33,174 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-13) [7246829] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:22:33,174 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-13) [7246829] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:22:33,184 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-13) [7246829] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:22:36,193 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:22:37,174 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:22:37,174 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:22:37,174 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-93) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:22:37,176 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-93) [4f020087] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:22:37,176 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-93) [4f020087] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:22:37,182 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-93) [4f020087] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:22:40,190 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:22:42,190 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:22:42,190 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:22:42,190 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-98) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:22:42,191 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-98) [eb8a062] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:22:42,191 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-98) [eb8a062] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:22:42,199 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-98) [eb8a062] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:22:45,205 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:22:46,192 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:22:46,192 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:22:46,192 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-85) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:22:46,194 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-85) [61bce53] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:22:46,195 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-85) [61bce53] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:22:46,205 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-85) [61bce53] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:22:49,213 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:22:51,213 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:22:51,213 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:22:51,213 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-81) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:22:51,215 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-81) [2dc3b30d] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:22:51,215 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-81) [2dc3b30d] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:22:51,224 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-81) [2dc3b30d] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:22:54,233 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:22:55,215 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:22:55,215 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:22:55,215 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-54) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:22:55,216 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-54) [15e2b976] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:22:55,216 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-54) [15e2b976] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:22:55,225 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-54) [15e2b976] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:22:58,233 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:23:00,233 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:23:00,233 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:23:00,233 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-80) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:23:00,235 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-80) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:23:00,235 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-80) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:23:00,243 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-80) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:23:03,251 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:23:04,235 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:23:04,235 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:23:04,235 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-43) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:23:04,236 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-43) [2d3b2827] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:23:04,236 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-43) [2d3b2827] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:23:04,243 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-43) [2d3b2827] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:23:07,252 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:23:09,252 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:23:09,252 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:23:09,252 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-94) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:23:09,260 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-94) [5dd0314d] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:23:09,260 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-94) [5dd0314d] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:23:09,267 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-94) [5dd0314d] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:23:12,273 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:23:13,254 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:23:13,254 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:23:13,254 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-91) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:23:13,256 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-91) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:23:13,256 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-91) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:23:13,263 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-91) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:23:16,273 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:23:18,273 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:23:18,273 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:23:18,273 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-73) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:23:18,274 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-73) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:23:18,274 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-73) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:23:18,287 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-73) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:23:21,295 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:23:22,275 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:23:22,275 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:23:22,275 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-68) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:23:22,276 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-68) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:23:22,277 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-68) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:23:22,284 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-68) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:23:25,292 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:23:27,292 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:23:27,292 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:23:27,292 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-5) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:23:27,293 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-5) [13a89c46] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:23:27,293 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-5) [13a89c46] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:23:27,301 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-5) [13a89c46] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:23:30,308 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:23:31,294 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:23:31,294 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:23:31,294 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-3) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:23:31,296 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-3) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:23:31,296 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-3) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:23:31,307 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-3) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:23:34,317 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:23:36,317 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:23:36,317 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:23:36,317 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-49) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:23:36,319 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-49) [196d3ba4] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:23:36,319 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-49) [196d3ba4] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:23:36,328 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-49) [196d3ba4] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:23:39,340 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:23:40,320 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:23:40,320 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:23:40,320 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-10) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:23:40,321 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-10) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:23:40,321 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-10) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:23:40,333 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-10) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:23:43,341 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:23:45,341 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:23:45,341 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:23:45,341 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-21) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:23:45,342 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-21) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:23:45,342 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-21) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:23:45,351 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-21) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:23:48,358 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:23:49,343 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:23:49,343 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:23:49,343 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-62) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:23:49,345 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-62) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:23:49,345 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-62) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:23:49,352 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-62) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:23:52,360 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:23:54,360 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:23:54,360 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:23:54,360 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-20) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:23:54,361 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-20) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:23:54,361 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-20) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:23:54,369 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-20) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:23:57,375 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:23:58,362 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:23:58,362 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:23:58,362 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-64) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:23:58,363 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-64) [785bb27f] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:23:58,363 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-64) [785bb27f] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:23:58,371 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-64) [785bb27f] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:24:01,379 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:24:03,379 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:24:03,379 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:24:03,379 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-28) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:24:03,380 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-28) [60adf93f] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:24:03,380 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-28) [60adf93f] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:24:03,389 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-28) [60adf93f] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:24:06,398 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:24:07,381 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:24:07,381 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:24:07,381 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-98) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:24:07,383 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-98) [eb8a062] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:24:07,383 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-98) [eb8a062] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:24:07,390 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-98) [eb8a062] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:24:10,399 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:24:12,399 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:24:12,399 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:24:12,399 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-5) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:24:12,400 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-5) [13a89c46] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:24:12,400 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-5) [13a89c46] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:24:12,408 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-5) [13a89c46] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:24:15,414 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:24:16,401 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:24:16,401 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:24:16,401 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-33) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:24:16,403 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-33) [451c3558] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:24:16,403 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-33) [451c3558] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:24:16,410 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-33) [451c3558] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:24:19,417 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:24:21,417 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:24:21,417 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:24:21,417 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-55) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:24:21,419 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-55) [7a4dea71] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:24:21,419 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-55) [7a4dea71] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:24:21,427 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-55) [7a4dea71] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:24:24,437 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:24:25,419 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:24:25,419 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:24:25,419 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-84) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:24:25,422 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-84) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:24:25,422 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-84) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:24:25,432 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-84) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:24:28,440 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:24:30,441 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:24:30,441 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:24:30,441 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-15) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:24:30,442 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-15) [3a4dd3fc] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:24:30,443 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-15) [3a4dd3fc] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:24:30,450 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-15) [3a4dd3fc] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:24:33,456 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:24:34,443 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:24:34,443 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:24:34,443 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-48) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:24:34,444 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-48) [40606675] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:24:34,444 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-48) [40606675] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:24:34,453 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-48) [40606675] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:24:37,462 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:24:39,462 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:24:39,462 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:24:39,462 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-51) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:24:39,463 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-51) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:24:39,463 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-51) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:24:39,471 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-51) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:24:42,478 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:24:43,466 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:24:43,466 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:24:43,466 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-63) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:24:43,468 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-63) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:24:43,468 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-63) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:24:43,475 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-63) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:24:46,484 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:24:48,484 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:24:48,484 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:24:48,484 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-18) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:24:48,485 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-18) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:24:48,485 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-18) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:24:48,494 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-18) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:24:51,500 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:24:52,486 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:24:52,486 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:24:52,486 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-59) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:24:52,487 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-59) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:24:52,487 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-59) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:24:52,494 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-59) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:24:55,501 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:24:57,501 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:24:57,501 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:24:57,501 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-75) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:24:57,502 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-75) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:24:57,502 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-75) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:24:57,509 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-75) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:25:00,517 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:25:01,503 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:25:01,503 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:25:01,503 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-1) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:25:01,505 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-1) [30db9a86] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:25:01,505 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-1) [30db9a86] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:25:01,512 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-1) [30db9a86] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:25:04,518 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:25:06,518 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:25:06,518 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:25:06,518 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-78) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:25:06,520 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-78) [4e91cc97] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:25:06,520 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-78) [4e91cc97] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:25:06,528 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-78) [4e91cc97] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:25:09,535 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:25:10,520 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:25:10,521 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:25:10,521 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-43) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:25:10,522 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-43) [2d3b2827] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:25:10,523 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-43) [2d3b2827] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:25:10,532 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-43) [2d3b2827] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:25:10,537 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-43) [2d3b2827] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:25:13,545 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:25:15,546 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:25:15,546 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:25:15,546 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-38) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:25:15,548 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-38) [56c37b5] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:25:15,548 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-38) [56c37b5] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:25:15,555 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-38) [56c37b5] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:25:18,562 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:25:19,548 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:25:19,548 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:25:19,548 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-91) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:25:19,550 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-91) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:25:19,550 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-91) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:25:19,557 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-91) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:25:22,564 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:25:24,565 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:25:24,565 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:25:24,565 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-88) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:25:24,567 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-88) [f89ebd8] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:25:24,567 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-88) [f89ebd8] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:25:24,574 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-88) [f89ebd8] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:25:27,583 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:25:28,566 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:25:28,566 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:25:28,566 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-31) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:25:28,568 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-31) [2eda49c0] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:25:28,569 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-31) [2eda49c0] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:25:28,579 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-31) [2eda49c0] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:25:31,588 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:25:33,589 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:25:33,589 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:25:33,589 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-11) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:25:33,591 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-11) [4df5b8de] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:25:33,591 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-11) [4df5b8de] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:25:33,598 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-11) [4df5b8de] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:25:36,605 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:25:37,591 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:25:37,591 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:25:37,591 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-16) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:25:37,593 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-16) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:25:37,593 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-16) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:25:37,601 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-16) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:25:40,608 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:25:42,608 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:25:42,608 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:25:42,608 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-33) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:25:42,609 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-33) [451c3558] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:25:42,609 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-33) [451c3558] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:25:42,618 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-33) [451c3558] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:25:45,627 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:25:46,610 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:25:46,610 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:25:46,610 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-70) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:25:46,612 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-70) [562b8b0f] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:25:46,612 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-70) [562b8b0f] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:25:46,621 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-70) [562b8b0f] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:25:49,630 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:25:51,630 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:25:51,630 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:25:51,630 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-53) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:25:51,632 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-53) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:25:51,632 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-53) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:25:51,640 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-53) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:25:54,648 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:25:55,632 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:25:55,632 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:25:55,632 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-26) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:25:55,634 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-26) [13b06b4d] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:25:55,634 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-26) [13b06b4d] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:25:55,645 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-26) [13b06b4d] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:25:58,652 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:26:00,653 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:26:00,653 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:26:00,653 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-7) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:26:00,654 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-7) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:26:00,654 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-7) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:26:00,662 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-7) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:26:03,669 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:26:04,661 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:26:04,661 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:26:04,661 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-97) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:26:04,663 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-97) [54081188] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:26:04,663 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-97) [54081188] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:26:04,671 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-97) [54081188] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:26:07,678 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:26:09,679 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:26:09,679 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:26:09,679 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-93) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:26:09,680 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-93) [4f020087] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:26:09,680 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-93) [4f020087] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:26:09,688 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-93) [4f020087] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:26:12,695 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:26:13,681 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:26:13,681 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:26:13,681 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-2) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:26:13,683 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-2) [16829b39] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:26:13,683 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-2) [16829b39] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:26:13,690 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-2) [16829b39] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:26:15,003 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp-/127.0.0.1:8702-8) [7d83eb05] Running command: LogoutUserCommand internal: false. >2014-10-08 09:26:15,003 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp-/127.0.0.1:8702-4) [7277e83c] Running command: LogoutUserCommand internal: false. >2014-10-08 09:26:15,021 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-4) [7277e83c] Correlation ID: 7277e83c, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-10-08 09:26:15,035 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-8) [7d83eb05] Correlation ID: 7d83eb05, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-10-08 09:26:15,141 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp-/127.0.0.1:8702-2) [1d2111a5] Running command: LogoutUserCommand internal: false. >2014-10-08 09:26:15,143 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp-/127.0.0.1:8702-5) [234839d8] Running command: LogoutUserCommand internal: false. >2014-10-08 09:26:15,144 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp-/127.0.0.1:8702-10) [12763f22] Running command: LogoutUserCommand internal: false. >2014-10-08 09:26:15,145 INFO [org.ovirt.engine.core.bll.aaa.LogoutUserCommand] (ajp-/127.0.0.1:8702-7) [e855a8a] Running command: LogoutUserCommand internal: false. >2014-10-08 09:26:15,150 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-2) [1d2111a5] Correlation ID: 1d2111a5, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-10-08 09:26:15,153 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-5) [234839d8] Correlation ID: 234839d8, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-10-08 09:26:15,155 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-7) [e855a8a] Correlation ID: e855a8a, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-10-08 09:26:15,155 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-10) [12763f22] Correlation ID: 12763f22, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. >2014-10-08 09:26:15,202 INFO [org.ovirt.engine.core.utils.servlet.UnsupportedLocaleHelper] (ajp-/127.0.0.1:8702-1) Invalid locale found in configuration: >2014-10-08 09:26:15,202 INFO [org.ovirt.engine.core.utils.servlet.UnsupportedLocaleHelper] (ajp-/127.0.0.1:8702-1) Invalid locale found in configuration: >2014-10-08 09:26:16,705 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:26:18,705 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:26:18,705 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:26:18,705 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-69) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:26:18,706 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-69) [294797be] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:26:18,706 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-69) [294797be] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:26:18,714 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-69) [294797be] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:26:21,721 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:26:22,709 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:26:22,709 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:26:22,709 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-29) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:26:22,711 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-29) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:26:22,711 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-29) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:26:22,733 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-29) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:26:25,740 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:26:27,740 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:26:27,740 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:26:27,740 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-40) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:26:27,743 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-40) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:26:27,743 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-40) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:26:27,753 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-40) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:26:30,761 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:26:31,742 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:26:31,742 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:26:31,742 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-80) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:26:31,745 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-80) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:26:31,745 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-80) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:26:31,755 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-80) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:26:34,764 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:26:36,764 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:26:36,764 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:26:36,764 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-46) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:26:36,766 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-46) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:26:36,766 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-46) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:26:36,773 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-46) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:26:39,782 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:26:40,766 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:26:40,766 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:26:40,766 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-34) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:26:40,767 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-34) [3e42dc0b] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:26:40,767 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-34) [3e42dc0b] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:26:40,774 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-34) [3e42dc0b] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:26:43,780 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:26:45,780 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:26:45,780 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:26:45,780 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-50) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:26:45,781 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-50) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:26:45,781 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-50) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:26:45,789 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-50) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:26:48,796 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:26:49,782 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:26:49,782 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:26:49,782 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-63) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:26:49,784 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-63) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:26:49,784 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-63) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:26:49,791 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-63) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:26:52,798 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:26:54,798 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:26:54,798 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:26:54,798 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-95) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:26:54,800 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-95) [4bd2250] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:26:54,800 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-95) [4bd2250] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:26:54,808 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-95) [4bd2250] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:26:57,814 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:26:58,800 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:26:58,800 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:26:58,800 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-59) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:26:58,802 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-59) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:26:58,802 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-59) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:26:58,811 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-59) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:27:01,821 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:27:03,821 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:27:03,821 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:27:03,821 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-85) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:27:03,823 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-85) [61bce53] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:27:03,823 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-85) [61bce53] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:27:03,832 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-85) [61bce53] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:27:06,843 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:27:07,823 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:27:07,823 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:27:07,823 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-60) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:27:07,824 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-60) [7bc6f040] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:27:07,824 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-60) [7bc6f040] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:27:07,833 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-60) [7bc6f040] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:27:10,839 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:27:12,839 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:27:12,839 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:27:12,839 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-40) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:27:12,841 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-40) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:27:12,841 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-40) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:27:12,849 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-40) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:27:15,857 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:27:16,841 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:27:16,841 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:27:16,841 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-14) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:27:16,843 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-14) [6343caea] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:27:16,843 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-14) [6343caea] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:27:16,854 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-14) [6343caea] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:27:19,862 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:27:21,862 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:27:21,862 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:27:21,862 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-45) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:27:21,864 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-45) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:27:21,864 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-45) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:27:21,873 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-45) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:27:24,883 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:27:25,864 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:27:25,864 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:27:25,864 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-22) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:27:25,866 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-22) [1e1b6a16] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:27:25,866 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-22) [1e1b6a16] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:27:25,874 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-22) [1e1b6a16] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:27:28,882 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:27:30,882 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:27:30,882 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:27:30,882 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-77) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:27:30,883 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-77) [22f3702] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:27:30,883 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-77) [22f3702] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:27:30,890 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-77) [22f3702] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:27:33,898 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:27:34,884 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:27:34,884 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:27:34,884 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-73) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:27:34,886 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-73) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:27:34,887 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-73) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:27:34,897 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-73) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:27:37,906 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:27:39,906 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:27:39,906 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:27:39,906 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-74) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:27:39,908 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-74) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:27:39,908 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-74) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:27:39,916 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-74) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:27:42,923 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:27:43,908 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:27:43,908 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:27:43,908 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-58) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:27:43,910 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-58) [1a19a2ca] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:27:43,910 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-58) [1a19a2ca] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:27:43,919 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-58) [1a19a2ca] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:27:46,928 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:27:48,928 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:27:48,928 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:27:48,928 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-81) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:27:48,929 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-81) [2dc3b30d] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:27:48,930 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-81) [2dc3b30d] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:27:48,937 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-81) [2dc3b30d] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:27:51,946 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:27:52,930 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:27:52,930 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:27:52,930 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-49) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:27:52,932 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-49) [196d3ba4] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:27:52,932 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-49) [196d3ba4] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:27:52,939 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-49) [196d3ba4] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:27:55,945 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:27:57,946 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:27:57,946 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:27:57,946 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-19) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:27:57,948 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-19) [5a26db10] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:27:57,948 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-19) [5a26db10] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:27:57,957 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-19) [5a26db10] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:28:00,963 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:28:01,948 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:28:01,948 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:28:01,948 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-71) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:28:01,950 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-71) [23e814fc] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:28:01,951 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-71) [23e814fc] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:28:01,961 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-71) [23e814fc] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:28:04,971 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:28:06,971 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:28:06,971 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:28:06,971 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-48) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:28:06,973 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-48) [40606675] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:28:06,973 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-48) [40606675] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:28:06,982 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-48) [40606675] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:28:09,989 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:28:10,973 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:28:10,973 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:28:10,973 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-51) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:28:10,975 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-51) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:28:10,975 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-51) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:28:10,982 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-51) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:28:10,987 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-51) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:28:13,995 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:28:15,996 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:28:15,996 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:28:15,996 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-99) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:28:15,997 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-99) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:28:15,997 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-99) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:28:16,005 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-99) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:28:19,011 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:28:19,998 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:28:19,998 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:28:19,998 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-31) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:28:20,000 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-31) [2eda49c0] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:28:20,000 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-31) [2eda49c0] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:28:20,008 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-31) [2eda49c0] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:28:23,016 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:28:25,017 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:28:25,017 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:28:25,017 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-35) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:28:25,018 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-35) [1ab5e2c1] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:28:25,018 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-35) [1ab5e2c1] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:28:25,026 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-35) [1ab5e2c1] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:28:28,032 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:28:29,018 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:28:29,018 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:28:29,018 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-85) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:28:29,020 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-85) [61bce53] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:28:29,020 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-85) [61bce53] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:28:29,027 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-85) [61bce53] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:28:32,035 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:28:34,035 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:28:34,035 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:28:34,035 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-6) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:28:34,036 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-6) [23c78bb] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:28:34,036 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-6) [23c78bb] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:28:34,044 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-6) [23c78bb] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:28:37,050 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:28:38,037 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:28:38,037 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:28:38,037 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-83) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:28:38,040 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-83) [6d722a1d] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:28:38,040 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-83) [6d722a1d] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:28:38,052 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-83) [6d722a1d] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:28:41,060 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:28:43,060 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:28:43,060 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:28:43,060 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-19) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:28:43,062 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-19) [5a26db10] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:28:43,062 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-19) [5a26db10] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:28:43,071 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-19) [5a26db10] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:28:46,078 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:28:47,062 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:28:47,062 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:28:47,062 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-26) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:28:47,064 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-26) [13b06b4d] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:28:47,064 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-26) [13b06b4d] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:28:47,076 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-26) [13b06b4d] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:28:50,083 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:28:52,083 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:28:52,083 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:28:52,083 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-38) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:28:52,085 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-38) [56c37b5] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:28:52,085 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-38) [56c37b5] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:28:52,093 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-38) [56c37b5] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:28:55,104 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:28:56,085 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:28:56,085 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:28:56,085 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-77) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:28:56,087 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-77) [22f3702] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:28:56,087 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-77) [22f3702] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:28:56,096 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-77) [22f3702] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:28:59,105 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:29:01,105 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:29:01,105 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:29:01,105 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-28) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:29:01,106 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-28) [60adf93f] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:29:01,106 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-28) [60adf93f] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:29:01,114 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-28) [60adf93f] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:29:04,122 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:29:05,107 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:29:05,107 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:29:05,107 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-87) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:29:05,110 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-87) [1dbf513e] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:29:05,110 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-87) [1dbf513e] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:29:05,121 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-87) [1dbf513e] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:29:08,130 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:29:10,130 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:29:10,130 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:29:10,130 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-58) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:29:10,131 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-58) [1a19a2ca] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:29:10,131 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-58) [1a19a2ca] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:29:10,139 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-58) [1a19a2ca] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:29:13,147 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:29:14,132 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:29:14,132 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:29:14,132 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-25) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:29:14,134 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-25) [5298f326] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:29:14,134 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-25) [5298f326] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:29:14,141 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-25) [5298f326] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:29:17,150 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:29:19,150 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:29:19,150 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:29:19,150 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-49) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:29:19,151 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-49) [196d3ba4] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:29:19,151 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-49) [196d3ba4] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:29:19,158 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-49) [196d3ba4] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:29:22,165 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:29:23,152 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:29:23,152 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:29:23,152 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-80) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:29:23,154 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-80) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:29:23,154 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-80) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:29:23,162 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-80) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:29:26,171 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:29:28,171 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:29:28,171 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:29:28,171 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-82) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:29:28,172 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-82) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:29:28,172 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-82) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:29:28,180 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-82) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:29:31,188 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:29:32,173 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:29:32,173 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:29:32,173 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-42) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:29:32,175 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-42) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:29:32,175 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-42) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:29:32,184 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-42) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:29:35,192 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:29:37,192 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:29:37,192 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:29:37,192 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-91) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:29:37,194 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-91) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:29:37,194 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-91) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:29:37,202 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-91) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:29:40,211 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:29:41,194 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:29:41,194 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:29:41,194 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-65) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:29:41,195 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-65) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:29:41,196 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-65) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:29:41,202 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-65) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:29:44,210 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:29:46,210 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:29:46,210 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:29:46,210 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-68) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:29:46,212 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-68) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:29:46,212 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-68) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:29:46,220 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-68) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:29:49,228 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:29:50,212 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:29:50,212 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:29:50,212 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-59) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:29:50,214 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-59) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:29:50,214 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-59) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:29:50,222 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-59) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:29:53,228 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:29:54,271 INFO [org.ovirt.engine.core.utils.servlet.UnsupportedLocaleHelper] (ajp-/127.0.0.1:8702-2) Invalid locale found in configuration: >2014-10-08 09:29:54,271 INFO [org.ovirt.engine.core.utils.servlet.UnsupportedLocaleHelper] (ajp-/127.0.0.1:8702-2) Invalid locale found in configuration: >2014-10-08 09:29:55,229 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:29:55,229 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:29:55,229 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-69) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:29:55,230 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-69) [294797be] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:29:55,230 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-69) [294797be] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:29:55,238 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-69) [294797be] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:29:58,245 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:29:59,231 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:29:59,231 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:29:59,231 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-33) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:29:59,233 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-33) [451c3558] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:29:59,233 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-33) [451c3558] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:29:59,240 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-33) [451c3558] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:30:02,247 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:30:04,247 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:30:04,247 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:30:04,247 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-10) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:30:04,248 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-10) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:30:04,248 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-10) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:30:04,258 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-10) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:30:07,266 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:30:08,249 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:30:08,249 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:30:08,249 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-43) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:30:08,251 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-43) [2d3b2827] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:30:08,251 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-43) [2d3b2827] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:30:08,258 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-43) [2d3b2827] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:30:11,277 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:30:13,278 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:30:13,278 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:30:13,278 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-34) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:30:13,279 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-34) [3e42dc0b] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:30:13,280 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-34) [3e42dc0b] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:30:13,287 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-34) [3e42dc0b] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:30:16,294 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:30:17,279 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:30:17,279 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:30:17,279 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-50) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:30:17,281 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-50) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:30:17,281 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-50) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:30:17,291 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-50) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:30:20,298 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:30:22,298 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:30:22,298 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:30:22,298 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-63) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:30:22,299 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-63) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:30:22,299 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-63) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:30:22,308 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-63) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:30:25,317 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:30:26,300 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:30:26,300 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:30:26,300 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-31) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:30:26,302 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-31) [2eda49c0] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:30:26,302 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-31) [2eda49c0] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:30:26,312 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-31) [2eda49c0] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:30:29,321 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:30:31,321 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:30:31,321 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:30:31,321 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-87) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:30:31,323 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-87) [1dbf513e] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:30:31,323 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-87) [1dbf513e] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:30:31,330 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-87) [1dbf513e] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:30:34,337 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:30:35,323 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:30:35,323 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:30:35,323 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-75) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:30:35,325 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-75) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:30:35,325 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-75) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:30:35,332 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-75) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:30:38,339 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:30:40,339 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:30:40,339 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:30:40,339 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-96) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:30:40,340 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-96) [721bb546] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:30:40,340 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-96) [721bb546] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:30:40,348 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-96) [721bb546] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:30:43,355 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:30:44,341 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:30:44,341 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:30:44,341 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-61) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:30:44,343 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-61) [20ef7a0c] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:30:44,343 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-61) [20ef7a0c] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:30:44,353 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-61) [20ef7a0c] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:30:47,362 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:30:49,363 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:30:49,363 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:30:49,363 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-4) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:30:49,364 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-4) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:30:49,364 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-4) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:30:49,372 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-4) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:30:52,380 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:30:53,365 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:30:53,365 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:30:53,365 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-62) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:30:53,366 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-62) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:30:53,366 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-62) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:30:53,374 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-62) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:30:56,380 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:30:58,380 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:30:58,380 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:30:58,380 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-34) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:30:58,382 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-34) [3e42dc0b] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:30:58,382 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-34) [3e42dc0b] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:30:58,389 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-34) [3e42dc0b] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:31:01,397 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:31:02,382 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:31:02,382 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:31:02,382 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-51) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:31:02,384 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-51) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:31:02,385 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-51) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:31:02,396 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-51) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:31:05,404 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:31:07,405 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:31:07,405 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:31:07,405 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-32) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:31:07,406 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-32) [56381f88] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:31:07,407 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-32) [56381f88] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:31:07,416 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-32) [56381f88] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:31:10,424 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:31:11,407 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:31:11,407 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:31:11,407 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-74) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:31:11,408 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-74) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:31:11,408 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-74) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:31:11,420 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-74) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:31:11,425 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-74) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:31:14,432 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:31:16,432 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:31:16,432 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:31:16,432 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-5) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:31:16,433 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-5) [13a89c46] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:31:16,433 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-5) [13a89c46] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:31:16,441 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-5) [13a89c46] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:31:19,447 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:31:20,436 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:31:20,436 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:31:20,436 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-66) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:31:20,437 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-66) [e582aa6] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:31:20,437 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-66) [e582aa6] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:31:20,445 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-66) [e582aa6] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:31:23,452 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:31:25,453 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:31:25,453 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:31:25,453 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-55) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:31:25,454 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-55) [7a4dea71] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:31:25,454 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-55) [7a4dea71] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:31:25,462 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-55) [7a4dea71] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:31:28,469 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:31:29,455 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:31:29,455 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:31:29,455 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-100) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:31:29,457 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-100) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:31:29,457 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-100) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:31:29,464 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-100) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:31:32,473 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:31:34,473 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:31:34,473 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:31:34,473 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-15) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:31:34,474 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-15) [3a4dd3fc] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:31:34,474 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-15) [3a4dd3fc] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:31:34,481 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-15) [3a4dd3fc] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:31:37,487 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:31:38,475 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:31:38,475 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:31:38,475 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-8) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:31:38,476 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-8) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:31:38,477 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-8) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:31:38,483 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-8) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:31:41,490 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:31:43,490 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:31:43,490 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:31:43,490 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-44) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:31:43,491 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-44) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:31:43,491 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-44) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:31:43,499 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-44) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:31:46,506 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:31:47,492 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:31:47,492 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:31:47,493 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-63) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:31:47,494 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-63) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:31:47,495 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-63) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:31:47,505 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-63) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:31:50,514 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:31:52,514 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:31:52,514 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:31:52,514 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-24) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:31:52,515 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-24) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:31:52,515 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-24) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:31:52,524 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-24) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:31:55,535 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:31:56,516 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:31:56,516 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:31:56,516 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-87) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:31:56,517 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-87) [1dbf513e] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:31:56,518 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-87) [1dbf513e] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:31:56,526 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-87) [1dbf513e] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:31:59,533 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:32:01,533 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:32:01,533 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:32:01,533 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-79) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:32:01,534 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-79) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:32:01,535 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-79) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:32:01,542 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-79) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:32:04,549 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:32:05,535 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:32:05,535 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:32:05,535 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-6) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:32:05,537 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-6) [23c78bb] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:32:05,537 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-6) [23c78bb] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:32:05,548 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-6) [23c78bb] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:32:08,555 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:32:10,555 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:32:10,555 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:32:10,555 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-61) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:32:10,556 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-61) [20ef7a0c] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:32:10,556 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-61) [20ef7a0c] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:32:10,565 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-61) [20ef7a0c] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:32:13,572 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:32:14,557 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:32:14,557 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:32:14,557 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-19) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:32:14,559 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-19) [5a26db10] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:32:14,559 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-19) [5a26db10] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:32:14,569 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-19) [5a26db10] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:32:17,579 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:32:19,579 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:32:19,581 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:32:19,581 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-17) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:32:19,583 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-17) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:32:19,583 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-17) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:32:19,590 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-17) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:32:22,597 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:32:23,582 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:32:23,582 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:32:23,582 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-86) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:32:23,583 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-86) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:32:23,584 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-86) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:32:23,592 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-86) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:32:26,600 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:32:28,600 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:32:28,600 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:32:28,600 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-77) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:32:28,601 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-77) [22f3702] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:32:28,602 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-77) [22f3702] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:32:28,610 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-77) [22f3702] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:32:31,616 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:32:32,602 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:32:32,602 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:32:32,602 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-47) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:32:32,605 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-47) [4028fe5f] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:32:32,605 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-47) [4028fe5f] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:32:32,616 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-47) [4028fe5f] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:32:35,624 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:32:37,625 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:32:37,625 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:32:37,625 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-98) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:32:37,626 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-98) [eb8a062] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:32:37,627 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-98) [eb8a062] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:32:37,634 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-98) [eb8a062] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:32:40,642 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:32:41,627 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:32:41,627 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:32:41,627 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-72) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:32:41,629 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-72) [513300b3] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:32:41,629 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-72) [513300b3] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:32:41,635 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-72) [513300b3] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:32:44,642 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:32:46,643 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:32:46,643 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:32:46,643 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-29) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:32:46,644 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-29) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:32:46,644 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-29) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:32:46,651 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-29) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:32:49,659 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:32:50,644 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:32:50,644 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:32:50,644 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-37) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:32:50,646 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-37) [310dcfa0] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:32:50,646 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-37) [310dcfa0] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:32:50,653 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-37) [310dcfa0] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:32:53,661 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:32:55,661 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:32:55,661 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:32:55,661 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-84) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:32:55,690 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-84) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:32:55,690 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-84) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:32:55,697 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-84) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:32:58,705 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:32:59,687 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:32:59,687 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:32:59,687 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-43) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:32:59,688 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-43) [2d3b2827] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:32:59,688 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-43) [2d3b2827] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:32:59,697 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-43) [2d3b2827] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:33:02,701 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:33:04,701 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:33:04,701 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:33:04,701 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-7) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:33:04,703 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-7) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:33:04,703 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-7) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:33:04,711 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-7) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:33:07,721 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:33:08,703 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:33:08,703 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:33:08,703 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-92) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:33:08,705 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-92) [3c53e142] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:33:08,706 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-92) [3c53e142] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:33:08,713 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-92) [3c53e142] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:33:11,719 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:33:13,720 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:33:13,720 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:33:13,720 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-88) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:33:13,721 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-88) [f89ebd8] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:33:13,721 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-88) [f89ebd8] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:33:13,728 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-88) [f89ebd8] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:33:16,735 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:33:17,721 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:33:17,721 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:33:17,722 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-68) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:33:17,724 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-68) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:33:17,724 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-68) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:33:17,735 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-68) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:33:20,744 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:33:22,745 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:33:22,745 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:33:22,745 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-98) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:33:22,747 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-98) [eb8a062] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:33:22,747 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-98) [eb8a062] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:33:22,758 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-98) [eb8a062] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:33:25,766 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:33:26,747 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:33:26,747 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:33:26,747 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-75) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:33:26,748 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-75) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:33:26,748 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-75) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:33:26,756 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-75) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:33:29,764 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:33:31,764 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:33:31,764 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:33:31,764 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-96) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:33:31,766 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-96) [721bb546] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:33:31,766 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-96) [721bb546] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:33:31,774 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-96) [721bb546] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:33:34,782 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:33:35,766 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:33:35,766 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:33:35,766 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-78) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:33:35,768 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-78) [4e91cc97] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:33:35,768 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-78) [4e91cc97] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:33:35,775 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-78) [4e91cc97] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:33:38,783 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:33:40,783 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:33:40,783 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:33:40,783 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-4) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:33:40,784 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-4) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:33:40,784 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-4) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:33:40,792 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-4) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:33:43,799 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:33:44,785 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:33:44,785 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:33:44,785 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-26) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:33:44,787 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-26) [13b06b4d] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:33:44,787 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-26) [13b06b4d] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:33:44,795 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-26) [13b06b4d] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:33:47,804 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:33:49,804 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:33:49,804 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:33:49,806 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-86) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:33:49,807 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-86) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:33:49,807 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-86) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:33:49,814 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-86) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:33:52,821 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:33:53,806 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:33:53,806 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:33:53,806 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-92) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:33:53,808 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-92) [3c53e142] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:33:53,808 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-92) [3c53e142] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:33:53,819 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-92) [3c53e142] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:33:56,827 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:33:58,827 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:33:58,827 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:33:58,827 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-88) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:33:58,828 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-88) [f89ebd8] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:33:58,828 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-88) [f89ebd8] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:33:58,835 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-88) [f89ebd8] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:34:01,843 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:34:02,829 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:34:02,829 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:34:02,829 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-89) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:34:02,831 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-89) [18caed52] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:34:02,831 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-89) [18caed52] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:34:02,841 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-89) [18caed52] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:34:05,851 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:34:07,851 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:34:07,851 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:34:07,851 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-58) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:34:07,853 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-58) [1a19a2ca] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:34:07,853 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-58) [1a19a2ca] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:34:07,864 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-58) [1a19a2ca] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:34:10,872 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:34:11,853 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:34:11,853 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:34:11,853 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-25) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:34:11,855 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-25) [5298f326] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:34:11,855 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-25) [5298f326] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:34:11,862 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-25) [5298f326] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:34:11,867 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-25) [5298f326] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:34:14,874 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:34:16,874 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:34:16,874 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:34:16,874 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-40) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:34:16,875 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-40) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:34:16,875 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-40) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:34:16,884 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-40) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:34:19,892 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:34:20,876 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:34:20,876 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:34:20,876 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-21) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:34:20,878 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-21) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:34:20,878 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-21) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:34:20,887 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-21) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:34:23,895 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:34:25,895 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:34:25,895 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:34:25,895 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-46) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:34:25,896 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-46) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:34:25,896 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-46) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:34:25,904 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-46) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:34:28,911 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:34:29,897 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:34:29,897 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:34:29,897 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-8) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:34:29,899 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-8) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:34:29,899 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-8) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:34:29,906 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-8) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:34:32,913 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:34:34,913 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:34:34,913 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:34:34,913 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-91) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:34:34,915 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-91) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:34:34,915 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-91) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:34:34,923 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-91) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:34:37,931 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:34:38,915 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:34:38,915 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:34:38,915 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-9) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:34:38,916 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-9) [5c6fc8e4] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:34:38,916 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-9) [5c6fc8e4] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:34:38,924 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-9) [5c6fc8e4] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:34:41,932 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:34:43,932 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:34:43,932 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:34:43,932 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-88) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:34:43,933 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-88) [f89ebd8] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:34:43,933 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-88) [f89ebd8] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:34:43,940 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-88) [f89ebd8] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:34:46,947 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:34:47,935 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:34:47,935 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:34:47,935 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-35) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:34:47,937 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-35) [1ab5e2c1] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:34:47,937 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-35) [1ab5e2c1] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:34:47,945 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-35) [1ab5e2c1] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:34:50,952 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:34:52,953 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:34:52,953 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:34:52,953 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-5) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:34:52,955 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-5) [13a89c46] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:34:52,955 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-5) [13a89c46] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:34:52,964 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-5) [13a89c46] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:34:55,978 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:34:56,955 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:34:56,955 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:34:56,955 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-54) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:34:56,957 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-54) [15e2b976] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:34:56,957 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-54) [15e2b976] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:34:56,965 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-54) [15e2b976] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:34:59,973 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:35:01,973 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:35:01,973 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:35:01,973 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-55) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:35:01,974 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-55) [7a4dea71] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:35:01,974 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-55) [7a4dea71] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:35:01,982 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-55) [7a4dea71] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:35:04,991 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:35:05,975 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:35:05,975 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:35:05,975 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-43) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:35:05,977 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-43) [2d3b2827] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:35:05,977 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-43) [2d3b2827] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:35:05,986 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-43) [2d3b2827] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:35:08,994 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:35:10,994 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:35:10,994 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:35:10,994 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-20) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:35:10,996 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-20) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:35:10,996 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-20) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:35:11,007 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-20) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:35:14,024 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:35:14,996 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:35:14,996 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:35:14,996 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-64) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:35:14,997 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-64) [785bb27f] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:35:14,997 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-64) [785bb27f] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:35:15,006 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-64) [785bb27f] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:35:18,014 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:35:20,015 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:35:20,015 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:35:20,017 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-63) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:35:20,018 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-63) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:35:20,018 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-63) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:35:20,025 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-63) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:35:23,032 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:35:24,017 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:35:24,017 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:35:24,017 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-52) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:35:24,019 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-52) [7f575cd6] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:35:24,019 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-52) [7f575cd6] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:35:24,026 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-52) [7f575cd6] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:35:27,033 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:35:29,034 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:35:29,034 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:35:29,034 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-36) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:35:29,035 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-36) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:35:29,036 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-36) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:35:29,043 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-36) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:35:32,049 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:35:33,036 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:35:33,036 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:35:33,036 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-16) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:35:33,037 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-16) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:35:33,038 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-16) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:35:33,046 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-16) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:35:36,052 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:35:38,052 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:35:38,052 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:35:38,052 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-60) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:35:38,053 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-60) [7bc6f040] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:35:38,054 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-60) [7bc6f040] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:35:38,061 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-60) [7bc6f040] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:35:41,070 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:35:42,054 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:35:42,054 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:35:42,054 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-67) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:35:42,056 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-67) [6cea7f79] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:35:42,056 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-67) [6cea7f79] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:35:42,065 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-67) [6cea7f79] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:35:45,074 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:35:47,075 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:35:47,075 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:35:47,075 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-71) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:35:47,076 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-71) [23e814fc] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:35:47,076 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-71) [23e814fc] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:35:47,084 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-71) [23e814fc] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:35:50,091 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:35:51,077 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:35:51,077 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:35:51,077 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-45) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:35:51,079 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-45) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:35:51,079 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-45) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:35:51,086 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-45) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:35:54,095 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:35:56,095 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:35:56,095 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:35:56,097 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-8) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:35:56,098 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-8) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:35:56,098 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-8) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:35:56,106 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-8) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:35:59,113 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:36:00,097 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:36:00,097 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:36:00,097 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-64) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:36:00,098 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-64) [785bb27f] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:36:00,099 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-64) [785bb27f] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:36:00,106 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-64) [785bb27f] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:36:03,113 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:36:05,113 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:36:05,113 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:36:05,113 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-63) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:36:05,114 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-63) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:36:05,114 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-63) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:36:05,122 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-63) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:36:08,131 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:36:09,115 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:36:09,115 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:36:09,115 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-59) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:36:09,116 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-59) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:36:09,117 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-59) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:36:09,124 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-59) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:36:12,131 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:36:14,131 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:36:14,131 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:36:14,131 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-36) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:36:14,132 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-36) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:36:14,132 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-36) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:36:14,140 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-36) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:36:17,149 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:36:18,133 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:36:18,133 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:36:18,133 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-33) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:36:18,135 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-33) [451c3558] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:36:18,135 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-33) [451c3558] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:36:18,145 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-33) [451c3558] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:36:21,153 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:36:23,153 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:36:23,153 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:36:23,153 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-37) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:36:23,156 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-37) [310dcfa0] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:36:23,156 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-37) [310dcfa0] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:36:23,166 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-37) [310dcfa0] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:36:26,176 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:36:27,155 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:36:27,155 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:36:27,155 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-100) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:36:27,158 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-100) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:36:27,158 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-100) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:36:27,167 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-100) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:36:30,174 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:36:32,175 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:36:32,175 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:36:32,175 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-71) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:36:32,177 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-71) [23e814fc] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:36:32,177 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-71) [23e814fc] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:36:32,184 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-71) [23e814fc] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:36:35,192 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:36:36,177 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:36:36,177 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:36:36,177 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-34) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:36:36,178 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-34) [3e42dc0b] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:36:36,178 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-34) [3e42dc0b] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:36:36,186 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-34) [3e42dc0b] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:36:39,193 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:36:41,194 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:36:41,194 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:36:41,194 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-50) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:36:41,195 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-50) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:36:41,196 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-50) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:36:41,203 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-50) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:36:44,211 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:36:45,197 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:36:45,197 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:36:45,197 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-28) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:36:45,198 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-28) [60adf93f] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:36:45,198 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-28) [60adf93f] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:36:45,207 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-28) [60adf93f] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:36:48,216 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:36:50,216 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:36:50,216 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:36:50,217 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-63) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:36:50,219 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-63) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:36:50,219 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-63) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:36:50,226 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-63) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:36:53,232 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:36:54,218 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:36:54,218 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:36:54,218 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-59) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:36:54,220 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-59) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:36:54,220 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-59) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:36:54,226 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-59) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:36:57,232 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:36:59,233 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:36:59,233 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:36:59,233 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-57) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:36:59,234 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-57) [16b48795] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:36:59,234 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-57) [16b48795] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:36:59,242 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-57) [16b48795] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:37:02,249 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:37:03,234 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:37:03,234 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:37:03,234 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-29) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:37:03,236 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-29) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:37:03,236 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-29) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:37:03,243 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-29) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:37:06,250 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:37:08,250 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:37:08,250 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:37:08,250 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-67) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:37:08,251 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-67) [6cea7f79] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:37:08,252 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-67) [6cea7f79] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:37:08,259 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-67) [6cea7f79] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:37:11,266 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:37:12,252 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:37:12,252 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:37:12,252 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-12) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:37:12,253 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-12) [7458e5a2] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:37:12,253 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-12) [7458e5a2] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:37:12,259 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-12) [7458e5a2] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:37:12,264 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-12) [7458e5a2] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:37:15,270 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:37:17,271 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:37:17,271 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:37:17,271 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-42) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:37:17,272 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-42) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:37:17,272 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-42) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:37:17,280 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-42) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:37:20,287 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:37:21,273 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:37:21,273 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:37:21,273 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-76) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:37:21,275 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-76) [1bdd13e4] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:37:21,275 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-76) [1bdd13e4] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:37:21,282 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-76) [1bdd13e4] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:37:24,288 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:37:26,289 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:37:26,289 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:37:26,289 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-64) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:37:26,290 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-64) [785bb27f] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:37:26,290 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-64) [785bb27f] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:37:26,298 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-64) [785bb27f] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:37:29,305 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:37:30,291 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:37:30,291 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:37:30,291 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-47) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:37:30,292 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-47) [4028fe5f] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:37:30,293 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-47) [4028fe5f] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:37:30,300 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-47) [4028fe5f] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:37:33,307 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:37:35,308 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:37:35,308 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:37:35,308 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-68) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:37:35,309 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-68) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:37:35,310 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-68) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:37:35,317 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-68) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:37:38,323 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:37:39,309 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:37:39,309 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:37:39,309 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-79) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:37:39,311 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-79) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:37:39,311 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-79) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:37:39,319 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-79) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:37:42,325 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:37:44,325 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:37:44,325 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:37:44,325 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-57) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:37:44,326 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-57) [16b48795] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:37:44,326 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-57) [16b48795] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:37:44,334 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-57) [16b48795] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:37:47,341 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:37:48,327 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:37:48,327 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:37:48,327 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-70) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:37:48,329 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-70) [562b8b0f] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:37:48,329 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-70) [562b8b0f] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:37:48,338 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-70) [562b8b0f] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:37:51,346 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:37:53,346 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:37:53,346 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:37:53,346 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-55) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:37:53,347 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-55) [7a4dea71] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:37:53,348 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-55) [7a4dea71] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:37:53,355 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-55) [7a4dea71] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:37:56,365 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:37:57,348 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:37:57,348 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:37:57,348 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-71) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:37:57,350 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-71) [23e814fc] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:37:57,350 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-71) [23e814fc] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:37:57,357 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-71) [23e814fc] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:38:00,365 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:38:02,366 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:38:02,366 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:38:02,366 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-26) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:38:02,367 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-26) [13b06b4d] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:38:02,367 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-26) [13b06b4d] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:38:02,374 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-26) [13b06b4d] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:38:05,381 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:38:06,368 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:38:06,368 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:38:06,368 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-50) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:38:06,370 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-50) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:38:06,370 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-50) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:38:06,378 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-50) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:38:09,388 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:38:11,388 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:38:11,388 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:38:11,388 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-73) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:38:11,390 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-73) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:38:11,390 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-73) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:38:11,440 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-73) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:38:14,448 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:38:15,390 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:38:15,390 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:38:15,390 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-63) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:38:15,392 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-63) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:38:15,392 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-63) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:38:15,403 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-63) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:38:18,413 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:38:20,413 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:38:20,413 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:38:20,413 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-59) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:38:20,415 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-59) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:38:20,415 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-59) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:38:20,424 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-59) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:38:23,435 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:38:24,415 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:38:24,415 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:38:24,415 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-81) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:38:24,417 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-81) [2dc3b30d] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:38:24,418 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-81) [2dc3b30d] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:38:24,427 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-81) [2dc3b30d] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:38:27,435 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:38:29,436 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:38:29,436 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:38:29,436 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-56) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:38:29,437 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-56) [60b15432] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:38:29,437 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-56) [60b15432] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:38:29,446 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-56) [60b15432] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:38:32,452 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:38:33,438 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:38:33,438 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:38:33,439 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-70) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:38:33,440 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-70) [562b8b0f] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:38:33,441 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-70) [562b8b0f] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:38:33,451 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-70) [562b8b0f] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:38:36,460 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:38:38,460 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:38:38,460 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:38:38,460 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-19) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:38:38,462 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-19) [5a26db10] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:38:38,462 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-19) [5a26db10] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:38:38,469 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-19) [5a26db10] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:38:41,479 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:38:42,462 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:38:42,462 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:38:42,462 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-46) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:38:42,463 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-46) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:38:42,464 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-46) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:38:42,471 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-46) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:38:45,479 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:38:47,480 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:38:47,480 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:38:47,480 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-30) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:38:47,481 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-30) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:38:47,481 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-30) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:38:47,489 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-30) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:38:50,496 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:38:51,482 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:38:51,482 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:38:51,482 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-50) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:38:51,483 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-50) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:38:51,483 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-50) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:38:51,491 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-50) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:38:54,500 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:38:56,500 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:38:56,500 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:38:56,500 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-73) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:38:56,502 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-73) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:38:56,502 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-73) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:38:56,509 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-73) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:38:59,515 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:39:00,502 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:39:00,502 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:39:00,502 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-89) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:39:00,503 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-89) [18caed52] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:39:00,503 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-89) [18caed52] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:39:00,511 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-89) [18caed52] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:39:03,520 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:39:05,520 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:39:05,520 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:39:05,520 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-59) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:39:05,521 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-59) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:39:05,521 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-59) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:39:05,531 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-59) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:39:08,538 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:39:09,522 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:39:09,522 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:39:09,522 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-33) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:39:09,524 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-33) [451c3558] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:39:09,524 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-33) [451c3558] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:39:09,533 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-33) [451c3558] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:39:12,539 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:39:14,539 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:39:14,539 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:39:14,539 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-56) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:39:14,541 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-56) [60b15432] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:39:14,541 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-56) [60b15432] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:39:14,549 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-56) [60b15432] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:39:17,556 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:39:18,541 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:39:18,541 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:39:18,542 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-55) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:39:18,544 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-55) [7a4dea71] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:39:18,545 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-55) [7a4dea71] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:39:18,556 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-55) [7a4dea71] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:39:21,566 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:39:23,566 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:39:23,566 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:39:23,566 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-82) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:39:23,568 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-82) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:39:23,568 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-82) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:39:23,579 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-82) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:39:26,589 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:39:27,568 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:39:27,568 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:39:27,568 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-7) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:39:27,569 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-7) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:39:27,569 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-7) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:39:27,576 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-7) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:39:30,583 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:39:32,584 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:39:32,584 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:39:32,584 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-30) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:39:32,585 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-30) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:39:32,585 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-30) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:39:32,592 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-30) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:39:35,600 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:39:36,586 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:39:36,586 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:39:36,586 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-32) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:39:36,587 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-32) [56381f88] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:39:36,587 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-32) [56381f88] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:39:36,595 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-32) [56381f88] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:39:39,602 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:39:41,602 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:39:41,602 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:39:41,602 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-24) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:39:41,604 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-24) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:39:41,604 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-24) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:39:41,611 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-24) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:39:44,619 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:39:45,604 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:39:45,604 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:39:45,604 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-23) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:39:45,606 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-23) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:39:45,606 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-23) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:39:45,614 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-23) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:39:48,624 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:39:50,624 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:39:50,624 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:39:50,626 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-59) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:39:50,628 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-59) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:39:50,628 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-59) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:39:50,636 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-59) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:39:53,644 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:39:54,626 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:39:54,626 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:39:54,626 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-33) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:39:54,628 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-33) [451c3558] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:39:54,628 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-33) [451c3558] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:39:54,635 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-33) [451c3558] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:39:57,643 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:39:59,644 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:39:59,644 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:39:59,644 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-40) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:39:59,646 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-40) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:39:59,646 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-40) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:39:59,655 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-40) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:40:02,660 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:40:03,646 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:40:03,646 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:40:03,646 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-55) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:40:03,647 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-55) [7a4dea71] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:40:03,648 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-55) [7a4dea71] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:40:03,661 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-55) [7a4dea71] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:40:06,668 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:40:08,669 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:40:08,669 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:40:08,669 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-20) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:40:08,670 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-20) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:40:08,670 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-20) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:40:08,678 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-20) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:40:11,687 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:40:12,672 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:40:12,672 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:40:12,672 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-13) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:40:12,673 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-13) [7246829] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:40:12,673 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-13) [7246829] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:40:12,680 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-13) [7246829] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:40:12,686 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-13) [7246829] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:40:15,693 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:40:17,693 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:40:17,693 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:40:17,693 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-28) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:40:17,694 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-28) [60adf93f] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:40:17,694 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-28) [60adf93f] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:40:17,702 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-28) [60adf93f] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:40:20,709 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:40:21,696 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:40:21,696 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:40:21,696 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-52) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:40:21,697 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-52) [7f575cd6] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:40:21,698 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-52) [7f575cd6] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:40:21,705 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-52) [7f575cd6] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:40:24,713 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:40:26,713 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:40:26,713 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:40:26,713 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-87) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:40:26,714 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-87) [1dbf513e] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:40:26,714 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-87) [1dbf513e] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:40:26,721 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-87) [1dbf513e] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:40:29,735 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:40:30,715 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:40:30,715 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:40:30,715 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-5) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:40:30,716 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-5) [13a89c46] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:40:30,717 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-5) [13a89c46] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:40:30,725 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-5) [13a89c46] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:40:33,732 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:40:35,732 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:40:35,732 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:40:35,732 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-54) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:40:35,733 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-54) [15e2b976] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:40:35,734 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-54) [15e2b976] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:40:35,742 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-54) [15e2b976] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:40:38,750 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:40:39,734 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:40:39,734 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:40:39,734 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-70) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:40:39,736 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-70) [562b8b0f] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:40:39,736 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-70) [562b8b0f] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:40:39,742 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-70) [562b8b0f] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:40:42,748 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:40:44,748 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:40:44,748 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:40:44,748 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-14) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:40:44,749 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-14) [6343caea] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:40:44,749 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-14) [6343caea] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:40:44,758 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-14) [6343caea] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:40:47,764 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:40:48,750 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:40:48,750 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:40:48,750 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-94) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:40:48,752 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-94) [5dd0314d] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:40:48,752 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-94) [5dd0314d] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:40:48,760 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-94) [5dd0314d] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:40:51,768 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:40:53,768 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:40:53,768 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:40:53,768 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-22) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:40:53,769 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-22) [1e1b6a16] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:40:53,769 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-22) [1e1b6a16] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:40:53,777 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-22) [1e1b6a16] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:40:56,787 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:40:57,770 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:40:57,770 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:40:57,770 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-50) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:40:57,772 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-50) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:40:57,772 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-50) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:40:57,780 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-50) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:41:00,789 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:41:02,789 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:41:02,789 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:41:02,789 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-93) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:41:02,790 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-93) [4f020087] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:41:02,790 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-93) [4f020087] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:41:02,798 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-93) [4f020087] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:41:05,807 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:41:06,791 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:41:06,791 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:41:06,791 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-3) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:41:06,793 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-3) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:41:06,793 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-3) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:41:06,802 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-3) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:41:09,811 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:41:11,811 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:41:11,811 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:41:11,811 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-58) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:41:11,812 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-58) [1a19a2ca] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:41:11,812 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-58) [1a19a2ca] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:41:11,820 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-58) [1a19a2ca] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:41:14,828 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:41:15,813 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:41:15,813 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:41:15,813 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-29) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:41:15,815 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-29) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:41:15,815 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-29) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:41:15,821 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-29) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:41:18,829 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:41:20,829 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:41:20,829 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:41:20,829 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-80) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:41:20,831 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-80) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:41:20,831 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-80) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:41:20,838 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-80) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:41:23,845 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:41:24,831 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:41:24,831 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:41:24,831 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-21) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:41:24,833 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-21) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:41:24,833 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-21) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:41:24,840 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-21) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:41:27,847 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:41:29,847 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:41:29,847 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:41:29,847 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-62) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:41:29,848 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-62) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:41:29,848 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-62) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:41:29,858 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-62) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:41:32,868 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:41:33,849 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:41:33,849 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:41:33,850 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-34) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:41:33,851 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-34) [3e42dc0b] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:41:33,852 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-34) [3e42dc0b] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:41:33,862 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-34) [3e42dc0b] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:41:36,870 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:41:38,870 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:41:38,870 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:41:38,870 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-13) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:41:38,871 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-13) [7246829] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:41:38,872 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-13) [7246829] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:41:38,880 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-13) [7246829] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:41:41,889 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:41:42,872 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:41:42,872 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:41:42,872 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-9) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:41:42,873 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-9) [5c6fc8e4] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:41:42,874 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-9) [5c6fc8e4] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:41:42,880 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-9) [5c6fc8e4] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:41:45,888 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:41:47,889 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:41:47,889 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:41:47,889 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-24) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:41:47,890 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-24) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:41:47,891 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-24) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:41:47,900 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-24) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:41:50,908 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:41:51,889 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:41:51,890 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:41:51,890 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-72) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:41:51,892 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-72) [513300b3] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:41:51,892 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-72) [513300b3] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:41:51,899 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-72) [513300b3] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:41:54,908 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:41:56,908 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:41:56,908 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:41:56,910 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-5) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:41:56,912 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-5) [13a89c46] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:41:56,912 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-5) [13a89c46] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:41:56,919 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-5) [13a89c46] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:41:59,925 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:42:00,910 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:42:00,910 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:42:00,910 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-29) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:42:00,912 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-29) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:42:00,912 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-29) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:42:00,919 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-29) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:42:03,928 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:42:05,928 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:42:05,928 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:42:05,928 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-78) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:42:05,929 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-78) [4e91cc97] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:42:05,929 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-78) [4e91cc97] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:42:05,937 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-78) [4e91cc97] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:42:08,944 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:42:09,930 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:42:09,930 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:42:09,930 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-14) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:42:09,932 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-14) [6343caea] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:42:09,932 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-14) [6343caea] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:42:09,939 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-14) [6343caea] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:42:12,947 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:42:14,947 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:42:14,947 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:42:14,947 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-48) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:42:14,948 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-48) [40606675] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:42:14,948 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-48) [40606675] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:42:14,955 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-48) [40606675] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:42:17,962 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:42:18,949 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:42:18,949 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:42:18,949 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-30) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:42:18,951 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-30) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:42:18,951 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-30) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:42:18,958 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-30) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:42:21,964 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:42:23,964 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:42:23,964 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:42:23,964 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-77) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:42:23,966 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-77) [22f3702] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:42:23,966 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-77) [22f3702] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:42:23,975 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-77) [22f3702] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:42:26,984 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:42:27,966 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:42:27,966 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:42:27,966 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-93) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:42:27,967 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-93) [4f020087] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:42:27,968 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-93) [4f020087] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:42:27,975 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-93) [4f020087] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:42:30,984 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:42:32,984 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:42:32,984 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:42:32,984 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-11) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:42:32,985 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-11) [4df5b8de] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:42:32,985 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-11) [4df5b8de] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:42:32,992 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-11) [4df5b8de] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:42:36,000 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:42:36,986 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:42:36,986 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:42:36,986 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-16) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:42:36,987 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-16) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:42:36,988 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-16) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:42:36,996 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-16) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:42:40,003 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:42:42,003 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:42:42,003 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:42:42,003 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-96) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:42:42,004 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-96) [721bb546] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:42:42,004 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-96) [721bb546] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:42:42,012 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-96) [721bb546] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:42:45,020 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:42:46,005 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:42:46,005 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:42:46,005 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-61) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:42:46,007 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-61) [20ef7a0c] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:42:46,007 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-61) [20ef7a0c] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:42:46,013 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-61) [20ef7a0c] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:42:49,022 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:42:51,022 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:42:51,022 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:42:51,024 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-78) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:42:51,025 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-78) [4e91cc97] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:42:51,025 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-78) [4e91cc97] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:42:51,032 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-78) [4e91cc97] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:42:54,042 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:42:55,024 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:42:55,024 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:42:55,024 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-43) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:42:55,026 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-43) [2d3b2827] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:42:55,026 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-43) [2d3b2827] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:42:55,035 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-43) [2d3b2827] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:42:58,044 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:43:00,044 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:43:00,044 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:43:00,044 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-34) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:43:00,046 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-34) [3e42dc0b] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:43:00,046 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-34) [3e42dc0b] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:43:00,055 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-34) [3e42dc0b] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:43:03,063 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:43:04,046 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:43:04,046 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:43:04,047 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-30) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:43:04,051 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-30) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:43:04,051 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-30) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:43:04,063 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-30) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:43:07,076 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:43:09,076 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:43:09,076 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:43:09,076 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-28) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:43:09,077 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-28) [60adf93f] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:43:09,077 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-28) [60adf93f] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:43:09,084 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-28) [60adf93f] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:43:12,092 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:43:13,078 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:43:13,078 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:43:13,078 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-74) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:43:13,079 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-74) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:43:13,079 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-74) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:43:13,087 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-74) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:43:13,091 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-74) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:43:16,099 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:43:18,099 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:43:18,099 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:43:18,099 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-85) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:43:18,100 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-85) [61bce53] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:43:18,101 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-85) [61bce53] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:43:18,109 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-85) [61bce53] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:43:21,142 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:43:22,101 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:43:22,101 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:43:22,101 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-59) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:43:22,102 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-59) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:43:22,102 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-59) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:43:22,109 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-59) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:43:25,116 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:43:27,116 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:43:27,116 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:43:27,116 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-83) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:43:27,117 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-83) [6d722a1d] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:43:27,117 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-83) [6d722a1d] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:43:27,124 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-83) [6d722a1d] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:43:30,131 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:43:31,118 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:43:31,118 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:43:31,118 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-67) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:43:31,119 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-67) [6cea7f79] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:43:31,120 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-67) [6cea7f79] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:43:31,133 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-67) [6cea7f79] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:43:34,141 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:43:36,141 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:43:36,141 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:43:36,141 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-41) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:43:36,152 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-41) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:43:36,152 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-41) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:43:36,159 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-41) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:43:39,166 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:43:40,143 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:43:40,143 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:43:40,143 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-42) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:43:40,145 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-42) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:43:40,145 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-42) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:43:40,155 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-42) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:43:43,162 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:43:45,163 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:43:45,163 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:43:45,163 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-8) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:43:45,165 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-8) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:43:45,165 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-8) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:43:45,173 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-8) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:43:48,180 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:43:49,165 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:43:49,165 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:43:49,165 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-64) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:43:49,166 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-64) [785bb27f] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:43:49,166 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-64) [785bb27f] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:43:49,174 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-64) [785bb27f] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:43:52,180 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:43:54,181 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:43:54,181 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:43:54,181 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-95) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:43:54,182 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-95) [4bd2250] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:43:54,183 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-95) [4bd2250] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:43:54,190 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-95) [4bd2250] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:43:57,199 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:43:58,183 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:43:58,183 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:43:58,183 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-98) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:43:58,185 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-98) [eb8a062] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:43:58,185 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-98) [eb8a062] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:43:58,196 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-98) [eb8a062] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:44:01,205 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:44:03,205 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:44:03,205 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:44:03,205 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-87) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:44:03,206 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-87) [1dbf513e] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:44:03,206 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-87) [1dbf513e] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:44:03,213 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-87) [1dbf513e] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:44:06,222 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:44:07,207 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:44:07,207 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:44:07,207 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-96) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:44:07,208 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-96) [721bb546] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:44:07,208 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-96) [721bb546] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:44:07,216 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-96) [721bb546] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:44:10,223 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:44:12,223 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:44:12,223 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:44:12,223 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-80) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:44:12,224 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-80) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:44:12,225 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-80) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:44:12,233 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-80) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:44:15,240 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:44:16,226 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:44:16,226 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:44:16,226 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-19) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:44:16,228 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-19) [5a26db10] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:44:16,228 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-19) [5a26db10] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:44:16,235 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-19) [5a26db10] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:44:19,243 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:44:21,243 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:44:21,243 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:44:21,243 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-62) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:44:21,245 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-62) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:44:21,245 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-62) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:44:21,252 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-62) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:44:24,260 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:44:25,247 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:44:25,247 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:44:25,247 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-38) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:44:25,248 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-38) [56c37b5] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:44:25,249 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-38) [56c37b5] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:44:25,255 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-38) [56c37b5] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:44:28,263 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:44:30,263 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:44:30,263 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:44:30,263 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-30) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:44:30,264 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-30) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:44:30,264 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-30) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:44:30,297 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-30) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:44:33,305 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:44:34,266 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:44:34,266 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:44:34,266 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-32) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:44:34,268 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-32) [56381f88] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:44:34,268 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-32) [56381f88] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:44:34,277 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-32) [56381f88] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:44:37,284 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:44:39,285 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:44:39,285 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:44:39,285 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-89) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:44:39,287 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-89) [18caed52] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:44:39,287 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-89) [18caed52] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:44:39,295 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-89) [18caed52] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:44:42,305 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:44:43,287 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:44:43,287 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:44:43,287 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-23) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:44:43,289 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-23) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:44:43,289 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-23) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:44:43,297 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-23) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:44:46,305 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:44:48,305 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:44:48,305 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:44:48,305 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-66) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:44:48,306 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-66) [e582aa6] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:44:48,306 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-66) [e582aa6] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:44:48,314 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-66) [e582aa6] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:44:51,321 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:44:52,307 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:44:52,307 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:44:52,307 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-96) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:44:52,308 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-96) [721bb546] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:44:52,308 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-96) [721bb546] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:44:52,315 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-96) [721bb546] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:44:55,323 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:44:57,323 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:44:57,323 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:44:57,323 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-10) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:44:57,324 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-10) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:44:57,324 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-10) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:44:57,333 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-10) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:45:00,340 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:45:01,325 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:45:01,325 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:45:01,325 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-19) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:45:01,327 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-19) [5a26db10] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:45:01,327 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-19) [5a26db10] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:45:01,335 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-19) [5a26db10] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:45:04,343 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:45:06,343 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:45:06,343 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:45:06,343 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-76) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:45:06,344 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-76) [1bdd13e4] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:45:06,344 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-76) [1bdd13e4] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:45:06,353 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-76) [1bdd13e4] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:45:09,360 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:45:10,345 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:45:10,345 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:45:10,345 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-65) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:45:10,347 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-65) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:45:10,347 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-65) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:45:10,355 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-65) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:45:13,363 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:45:15,363 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:45:15,363 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:45:15,363 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-18) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:45:15,364 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-18) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:45:15,364 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-18) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:45:15,373 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-18) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:45:18,379 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:45:19,365 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:45:19,365 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:45:19,365 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-74) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:45:19,367 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-74) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:45:19,367 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-74) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:45:19,377 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-74) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:45:22,385 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:45:24,385 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:45:24,385 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:45:24,385 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-11) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:45:24,387 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-11) [4df5b8de] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:45:24,387 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-11) [4df5b8de] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:45:24,396 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-11) [4df5b8de] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:45:27,404 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:45:28,387 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:45:28,387 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:45:28,387 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-59) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:45:28,388 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-59) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:45:28,388 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-59) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:45:28,397 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-59) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:45:31,405 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:45:33,405 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:45:33,405 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:45:33,405 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-33) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:45:33,406 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-33) [451c3558] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:45:33,406 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-33) [451c3558] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:45:33,414 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-33) [451c3558] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:45:36,422 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:45:37,407 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:45:37,407 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:45:37,407 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-67) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:45:37,409 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-67) [6cea7f79] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:45:37,409 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-67) [6cea7f79] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:45:37,417 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-67) [6cea7f79] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:45:40,423 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:45:42,424 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:45:42,424 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:45:42,425 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-71) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:45:42,427 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-71) [23e814fc] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:45:42,427 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-71) [23e814fc] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:45:42,434 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-71) [23e814fc] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:45:45,442 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:45:46,426 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:45:46,426 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:45:46,426 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-7) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:45:46,427 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-7) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:45:46,427 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-7) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:45:46,435 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-7) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:45:49,444 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:45:51,444 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:45:51,444 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:45:51,444 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-76) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:45:51,445 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-76) [1bdd13e4] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:45:51,445 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-76) [1bdd13e4] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:45:51,453 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-76) [1bdd13e4] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:45:54,462 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:45:55,446 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:45:55,446 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:45:55,446 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-50) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:45:55,448 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-50) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:45:55,448 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-50) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:45:55,455 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-50) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:45:58,463 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:46:00,463 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:46:00,463 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:46:00,465 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-28) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:46:00,467 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-28) [60adf93f] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:46:00,467 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-28) [60adf93f] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:46:00,474 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-28) [60adf93f] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:46:03,482 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:46:04,465 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:46:04,465 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:46:04,465 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-74) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:46:04,467 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-74) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:46:04,468 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-74) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:46:04,478 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-74) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:46:07,487 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:46:09,487 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:46:09,487 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:46:09,487 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-58) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:46:09,488 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-58) [1a19a2ca] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:46:09,489 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-58) [1a19a2ca] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:46:09,497 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-58) [1a19a2ca] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:46:12,515 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:46:13,489 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:46:13,489 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:46:13,489 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-57) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:46:13,491 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-57) [16b48795] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:46:13,491 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-57) [16b48795] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:46:13,497 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-57) [16b48795] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:46:13,503 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-57) [16b48795] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:46:16,511 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:46:18,512 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:46:18,512 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:46:18,512 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-40) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:46:18,516 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-40) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:46:18,516 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-40) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:46:18,526 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-40) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:46:21,534 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:46:22,513 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:46:22,513 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:46:22,513 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-4) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:46:22,516 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-4) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:46:22,516 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-4) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:46:22,523 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-4) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:46:25,531 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:46:27,532 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:46:27,532 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:46:27,532 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-45) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:46:27,533 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-45) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:46:27,533 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-45) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:46:27,540 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-45) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:46:30,547 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:46:31,534 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:46:31,534 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:46:31,534 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-34) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:46:31,536 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-34) [3e42dc0b] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:46:31,536 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-34) [3e42dc0b] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:46:31,543 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-34) [3e42dc0b] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:46:34,551 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:46:36,552 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:46:36,552 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:46:36,552 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-92) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:46:36,553 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-92) [3c53e142] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:46:36,553 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-92) [3c53e142] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:46:36,563 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-92) [3c53e142] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:46:39,571 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:46:40,555 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:46:40,555 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:46:40,555 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-47) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:46:40,556 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-47) [4028fe5f] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:46:40,557 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-47) [4028fe5f] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:46:40,564 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-47) [4028fe5f] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:46:43,571 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:46:45,571 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:46:45,571 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:46:45,571 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-52) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:46:45,573 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-52) [7f575cd6] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:46:45,573 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-52) [7f575cd6] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:46:45,582 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-52) [7f575cd6] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:46:48,590 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:46:49,573 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:46:49,573 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:46:49,573 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-23) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:46:49,575 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-23) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:46:49,575 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-23) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:46:49,582 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-23) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:46:52,589 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:46:54,589 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:46:54,589 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:46:54,592 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-85) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:46:54,593 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-85) [61bce53] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:46:54,593 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-85) [61bce53] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:46:54,602 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-85) [61bce53] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:46:57,611 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:46:58,591 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:46:58,591 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:46:58,591 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-96) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:46:58,593 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-96) [721bb546] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:46:58,593 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-96) [721bb546] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:46:58,603 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-96) [721bb546] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:47:01,611 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:47:03,611 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:47:03,611 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:47:03,611 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-56) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:47:03,612 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-56) [60b15432] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:47:03,612 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-56) [60b15432] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:47:03,620 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-56) [60b15432] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:47:06,627 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:47:07,613 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:47:07,613 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:47:07,613 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-55) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:47:07,614 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-55) [7a4dea71] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:47:07,614 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-55) [7a4dea71] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:47:07,622 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-55) [7a4dea71] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:47:10,630 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:47:12,630 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:47:12,630 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:47:12,631 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-17) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:47:12,633 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-17) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:47:12,633 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-17) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:47:12,641 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-17) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:47:15,648 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:47:16,632 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:47:16,632 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:47:16,632 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-91) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:47:16,634 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-91) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:47:16,634 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-91) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:47:16,644 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-91) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:47:19,654 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:47:21,655 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:47:21,655 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:47:21,655 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-99) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:47:21,657 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-99) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:47:21,657 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-99) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:47:21,665 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-99) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:47:24,673 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:47:25,657 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:47:25,657 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:47:25,657 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-28) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:47:25,659 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-28) [60adf93f] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:47:25,659 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-28) [60adf93f] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:47:25,666 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-28) [60adf93f] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:47:28,674 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:47:30,674 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:47:30,674 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:47:30,676 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-3) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:47:30,688 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-3) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:47:30,688 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-3) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:47:30,696 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-3) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:47:33,701 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:47:34,676 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:47:34,676 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:47:34,676 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-75) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:47:34,678 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-75) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:47:34,678 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-75) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:47:34,687 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-75) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:47:37,695 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:47:39,695 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:47:39,695 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:47:39,695 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-6) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:47:39,696 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-6) [23c78bb] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:47:39,697 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-6) [23c78bb] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:47:39,704 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-6) [23c78bb] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:47:42,713 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:47:43,697 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:47:43,697 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:47:43,697 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-37) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:47:43,699 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-37) [310dcfa0] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:47:43,699 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-37) [310dcfa0] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:47:43,707 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-37) [310dcfa0] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:47:46,715 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:47:48,715 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:47:48,715 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:47:48,715 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-4) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:47:48,717 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-4) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:47:48,717 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-4) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:47:48,725 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-4) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:47:51,733 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:47:52,717 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:47:52,717 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:47:52,717 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-55) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:47:52,718 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-55) [7a4dea71] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:47:52,718 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-55) [7a4dea71] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:47:52,727 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-55) [7a4dea71] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:47:55,735 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:47:57,735 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:47:57,735 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:47:57,735 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-42) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:47:57,736 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-42) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:47:57,736 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-42) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:47:57,744 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-42) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:48:00,750 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:48:01,738 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:48:01,738 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:48:01,738 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-97) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:48:01,740 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-97) [54081188] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:48:01,740 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-97) [54081188] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:48:01,748 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-97) [54081188] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:48:04,755 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:48:06,755 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:48:06,755 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:48:06,755 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-31) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:48:06,756 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-31) [2eda49c0] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:48:06,756 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-31) [2eda49c0] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:48:06,767 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-31) [2eda49c0] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:48:09,775 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:48:10,758 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:48:10,758 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:48:10,758 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-68) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:48:10,759 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-68) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:48:10,760 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-68) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:48:10,767 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-68) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:48:13,774 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:48:15,776 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:48:15,776 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:48:15,776 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-69) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:48:15,777 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-69) [294797be] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:48:15,777 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-69) [294797be] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:48:15,786 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-69) [294797be] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:48:18,794 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:48:19,778 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:48:19,778 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:48:19,778 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-85) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:48:19,780 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-85) [61bce53] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:48:19,780 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-85) [61bce53] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:48:19,787 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-85) [61bce53] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:48:22,794 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:48:24,795 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:48:24,795 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:48:24,795 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-29) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:48:24,796 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-29) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:48:24,796 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-29) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:48:24,804 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-29) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:48:27,813 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:48:28,797 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:48:28,797 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:48:28,799 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-84) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:48:28,801 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-84) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:48:28,801 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-84) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:48:28,808 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-84) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:48:31,815 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:48:33,815 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:48:33,815 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:48:33,815 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-21) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:48:33,816 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-21) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:48:33,816 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-21) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:48:33,823 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-21) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:48:36,830 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:48:37,817 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:48:37,817 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:48:37,817 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-7) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:48:37,818 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-7) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:48:37,819 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-7) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:48:37,826 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-7) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:48:40,834 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:48:42,834 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:48:42,834 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:48:42,834 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-38) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:48:42,836 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-38) [56c37b5] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:48:42,836 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-38) [56c37b5] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:48:42,844 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-38) [56c37b5] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:48:45,851 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:48:46,837 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:48:46,837 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:48:46,837 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-77) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:48:46,839 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-77) [22f3702] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:48:46,839 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-77) [22f3702] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:48:46,849 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-77) [22f3702] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:48:49,858 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:48:51,858 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:48:51,858 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:48:51,858 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-18) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:48:51,859 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-18) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:48:51,859 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-18) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:48:51,867 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-18) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:48:54,880 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:48:55,861 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:48:55,861 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:48:55,861 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-98) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:48:55,862 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-98) [eb8a062] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:48:55,862 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-98) [eb8a062] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:48:55,870 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-98) [eb8a062] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:48:58,877 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:49:00,877 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:49:00,877 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:49:00,877 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-16) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:49:00,878 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-16) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:49:00,878 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-16) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:49:00,886 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-16) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:49:03,894 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:49:04,879 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:49:04,879 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:49:04,879 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-60) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:49:04,880 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-60) [7bc6f040] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:49:04,881 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-60) [7bc6f040] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:49:04,887 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-60) [7bc6f040] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:49:07,894 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:49:09,894 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:49:09,894 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:49:09,896 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-29) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:49:09,898 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-29) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:49:09,898 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-29) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:49:09,907 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-29) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:49:12,918 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:49:13,896 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:49:13,896 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:49:13,896 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-100) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:49:13,898 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-100) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:49:13,898 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-100) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:49:13,906 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-100) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:49:13,910 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-100) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:49:16,918 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:49:18,918 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:49:18,918 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:49:18,918 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-43) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:49:18,919 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-43) [2d3b2827] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:49:18,919 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-43) [2d3b2827] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:49:18,926 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-43) [2d3b2827] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:49:21,933 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:49:22,920 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:49:22,920 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:49:22,920 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-48) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:49:22,922 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-48) [40606675] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:49:22,922 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-48) [40606675] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:49:22,932 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-48) [40606675] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:49:25,940 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:49:27,940 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:49:27,940 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:49:27,940 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-13) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:49:27,941 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-13) [7246829] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:49:27,942 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-13) [7246829] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:49:27,948 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-13) [7246829] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:49:30,956 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:49:31,943 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:49:31,943 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:49:31,943 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-9) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:49:31,945 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-9) [5c6fc8e4] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:49:31,945 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-9) [5c6fc8e4] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:49:31,954 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-9) [5c6fc8e4] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:49:34,962 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:49:36,962 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:49:36,962 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:49:36,962 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-93) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:49:36,969 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-93) [4f020087] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:49:36,969 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-93) [4f020087] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:49:36,977 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-93) [4f020087] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:49:39,985 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:49:40,964 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:49:40,964 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:49:40,964 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-36) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:49:40,967 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-36) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:49:40,967 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-36) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:49:40,975 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-36) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:49:43,983 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:49:45,983 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:49:45,983 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:49:45,983 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-58) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:49:45,984 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-58) [1a19a2ca] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:49:45,984 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-58) [1a19a2ca] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:49:45,993 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-58) [1a19a2ca] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:49:49,000 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:49:49,985 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:49:49,985 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:49:49,985 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-96) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:49:49,987 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-96) [721bb546] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:49:49,987 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-96) [721bb546] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:49:49,994 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-96) [721bb546] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:49:53,001 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:49:55,002 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:49:55,002 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:49:55,004 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-40) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:49:55,006 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-40) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:49:55,006 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-40) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:49:55,013 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-40) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:49:58,023 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:49:59,004 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:49:59,004 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:49:59,004 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-14) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:49:59,006 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-14) [6343caea] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:49:59,006 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-14) [6343caea] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:49:59,013 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-14) [6343caea] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:50:02,019 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:50:04,019 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:50:04,019 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:50:04,019 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-94) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:50:04,020 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-94) [5dd0314d] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:50:04,020 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-94) [5dd0314d] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:50:04,027 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-94) [5dd0314d] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:50:07,034 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:50:08,021 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:50:08,021 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:50:08,021 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-30) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:50:08,023 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-30) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:50:08,023 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-30) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:50:08,033 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-30) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:50:11,041 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:50:13,041 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:50:13,041 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:50:13,043 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-9) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:50:13,044 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-9) [5c6fc8e4] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:50:13,044 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-9) [5c6fc8e4] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:50:13,052 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-9) [5c6fc8e4] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:50:16,059 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:50:17,043 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:50:17,043 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:50:17,043 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-68) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:50:17,045 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-68) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:50:17,045 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-68) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:50:17,051 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-68) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:50:20,060 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:50:22,060 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:50:22,060 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:50:22,060 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-23) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:50:22,062 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-23) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:50:22,062 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-23) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:50:22,069 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-23) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:50:25,077 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:50:26,062 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:50:26,062 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:50:26,062 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-59) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:50:26,064 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-59) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:50:26,064 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-59) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:50:26,072 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-59) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:50:29,080 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:50:31,081 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:50:31,081 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:50:31,083 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-33) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:50:31,084 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-33) [451c3558] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:50:31,084 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-33) [451c3558] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:50:31,091 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-33) [451c3558] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:50:34,098 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:50:35,083 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:50:35,084 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:50:35,084 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-67) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:50:35,086 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-67) [6cea7f79] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:50:35,086 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-67) [6cea7f79] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:50:35,096 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-67) [6cea7f79] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:50:38,105 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:50:40,105 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:50:40,105 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:50:40,105 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-14) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:50:40,106 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-14) [6343caea] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:50:40,106 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-14) [6343caea] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:50:40,114 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-14) [6343caea] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:50:43,122 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:50:44,107 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:50:44,107 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:50:44,107 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-7) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:50:44,109 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-7) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:50:44,109 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-7) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:50:44,120 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-7) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:50:47,129 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:50:49,129 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:50:49,129 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:50:49,129 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-86) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:50:49,131 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-86) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:50:49,131 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-86) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:50:49,138 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-86) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:50:52,145 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:50:53,131 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:50:53,131 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:50:53,131 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-50) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:50:53,133 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-50) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:50:53,133 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-50) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:50:53,141 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-50) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:50:56,149 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:50:58,149 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:50:58,149 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:50:58,149 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-2) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:50:58,150 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-2) [16829b39] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:50:58,150 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-2) [16829b39] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:50:58,157 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-2) [16829b39] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:51:01,165 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:51:02,151 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:51:02,151 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:51:02,151 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-93) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:51:02,153 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-93) [4f020087] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:51:02,153 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-93) [4f020087] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:51:02,162 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-93) [4f020087] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:51:05,170 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:51:07,170 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:51:07,170 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:51:07,170 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-25) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:51:07,171 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-25) [5298f326] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:51:07,171 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-25) [5298f326] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:51:07,180 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-25) [5298f326] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:51:10,188 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:51:11,172 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:51:11,172 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:51:11,172 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-49) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:51:11,174 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-49) [196d3ba4] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:51:11,174 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-49) [196d3ba4] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:51:11,181 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-49) [196d3ba4] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:51:14,190 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:51:16,190 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:51:16,190 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:51:16,190 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-53) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:51:16,192 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-53) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:51:16,192 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-53) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:51:16,199 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-53) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:51:19,205 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:51:20,192 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:51:20,192 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:51:20,192 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-71) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:51:20,194 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-71) [23e814fc] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:51:20,194 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-71) [23e814fc] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:51:20,201 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-71) [23e814fc] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:51:23,209 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:51:25,209 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:51:25,209 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:51:25,211 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-46) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:51:25,213 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-46) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:51:25,213 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-46) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:51:25,221 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-46) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:51:28,230 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:51:29,211 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:51:29,211 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:51:29,211 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-51) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:51:29,213 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-51) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:51:29,213 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-51) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:51:29,220 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-51) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:51:32,227 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:51:34,228 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:51:34,228 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:51:34,228 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-91) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:51:34,229 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-91) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:51:34,229 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-91) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:51:34,236 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-91) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:51:37,244 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:51:38,230 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:51:38,230 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:51:38,230 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-31) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:51:38,232 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-31) [2eda49c0] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:51:38,232 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-31) [2eda49c0] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:51:38,239 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-31) [2eda49c0] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:51:41,247 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:51:43,247 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:51:43,247 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:51:43,247 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-24) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:51:43,248 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-24) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:51:43,249 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-24) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:51:43,256 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-24) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:51:46,264 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:51:47,249 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:51:47,249 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:51:47,249 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-11) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:51:47,251 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-11) [4df5b8de] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:51:47,251 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-11) [4df5b8de] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:51:47,259 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-11) [4df5b8de] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:51:50,267 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:51:52,267 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:51:52,267 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:51:52,267 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-85) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:51:52,269 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-85) [61bce53] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:51:52,269 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-85) [61bce53] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:51:52,277 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-85) [61bce53] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:51:55,284 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:51:56,269 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:51:56,269 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:51:56,269 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-54) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:51:56,271 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-54) [15e2b976] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:51:56,271 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-54) [15e2b976] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:51:56,278 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-54) [15e2b976] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:51:59,285 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:52:01,285 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:52:01,285 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:52:01,285 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-84) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:52:01,286 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-84) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:52:01,287 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-84) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:52:01,293 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-84) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:52:04,301 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:52:05,287 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:52:05,287 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:52:05,287 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-41) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:52:05,289 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-41) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:52:05,289 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-41) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:52:05,301 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-41) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:52:08,310 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:52:10,310 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:52:10,310 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:52:10,310 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-34) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:52:10,312 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-34) [3e42dc0b] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:52:10,312 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-34) [3e42dc0b] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:52:10,320 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-34) [3e42dc0b] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:52:13,330 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:52:14,312 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:52:14,312 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:52:14,312 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-97) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:52:14,314 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-97) [54081188] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:52:14,314 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-97) [54081188] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:52:14,319 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-97) [54081188] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:52:14,323 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-97) [54081188] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:52:17,330 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:52:19,330 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:52:19,330 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:52:19,330 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-64) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:52:19,331 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-64) [785bb27f] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:52:19,332 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-64) [785bb27f] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:52:19,338 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-64) [785bb27f] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:52:22,344 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:52:23,332 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:52:23,332 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:52:23,332 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-31) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:52:23,334 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-31) [2eda49c0] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:52:23,334 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-31) [2eda49c0] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:52:23,342 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-31) [2eda49c0] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:52:26,349 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:52:28,350 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:52:28,350 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:52:28,350 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-3) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:52:28,351 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-3) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:52:28,351 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-3) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:52:28,358 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-3) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:52:31,364 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:52:32,352 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:52:32,352 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:52:32,352 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-69) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:52:32,354 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-69) [294797be] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:52:32,354 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-69) [294797be] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:52:32,361 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-69) [294797be] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:52:35,369 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:52:37,369 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:52:37,369 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:52:37,369 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-39) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:52:37,370 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-39) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:52:37,370 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-39) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:52:37,378 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-39) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:52:40,386 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:52:41,372 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:52:41,372 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:52:41,372 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-70) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:52:41,374 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-70) [562b8b0f] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:52:41,374 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-70) [562b8b0f] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:52:41,382 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-70) [562b8b0f] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:52:44,389 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:52:46,389 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:52:46,389 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:52:46,389 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-67) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:52:46,390 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-67) [6cea7f79] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:52:46,390 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-67) [6cea7f79] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:52:46,398 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-67) [6cea7f79] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:52:49,406 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:52:50,391 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:52:50,391 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:52:50,391 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-26) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:52:50,393 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-26) [13b06b4d] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:52:50,393 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-26) [13b06b4d] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:52:50,402 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-26) [13b06b4d] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:52:53,410 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:52:55,411 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:52:55,411 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:52:55,411 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-42) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:52:55,414 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-42) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:52:55,414 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-42) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:52:55,424 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-42) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:52:58,433 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:52:59,413 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:52:59,413 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:52:59,413 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-30) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:52:59,415 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-30) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:52:59,415 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-30) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:52:59,423 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-30) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:53:02,431 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:53:04,431 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:53:04,431 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:53:04,431 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-47) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:53:04,432 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-47) [4028fe5f] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:53:04,433 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-47) [4028fe5f] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:53:04,440 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-47) [4028fe5f] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:53:07,452 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:53:08,433 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:53:08,433 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:53:08,433 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-74) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:53:08,435 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-74) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:53:08,435 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-74) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:53:08,445 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-74) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:53:11,453 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:53:13,453 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:53:13,453 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:53:13,453 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-3) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:53:13,454 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-3) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:53:13,454 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-3) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:53:13,461 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-3) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:53:16,468 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:53:17,456 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:53:17,456 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:53:17,456 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-59) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:53:17,457 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-59) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:53:17,457 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-59) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:53:17,465 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-59) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:53:20,473 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:53:22,474 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:53:22,474 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:53:22,474 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-33) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:53:22,476 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-33) [451c3558] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:53:22,476 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-33) [451c3558] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:53:22,486 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-33) [451c3558] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:53:25,495 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:53:26,475 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:53:26,475 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:53:26,475 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-12) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:53:26,477 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-12) [7458e5a2] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:53:26,477 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-12) [7458e5a2] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:53:26,486 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-12) [7458e5a2] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:53:29,494 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:53:31,494 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:53:31,494 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:53:31,494 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-21) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:53:31,496 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-21) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:53:31,496 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-21) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:53:31,503 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-21) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:53:34,511 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:53:35,497 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:53:35,497 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:53:35,497 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-48) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:53:35,499 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-48) [40606675] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:53:35,499 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-48) [40606675] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:53:35,507 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-48) [40606675] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:53:38,515 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:53:40,516 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:53:40,516 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:53:40,516 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-86) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:53:40,518 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-86) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:53:40,518 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-86) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:53:40,526 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-86) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:53:43,535 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:53:44,517 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:53:44,517 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:53:44,517 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-91) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:53:44,519 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-91) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:53:44,519 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-91) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:53:44,527 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-91) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:53:47,535 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:53:49,535 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:53:49,535 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:53:49,535 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-90) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:53:49,536 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-90) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:53:49,536 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-90) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:53:49,544 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-90) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:53:52,551 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:53:53,537 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:53:53,537 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:53:53,537 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-98) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:53:53,539 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-98) [eb8a062] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:53:53,539 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-98) [eb8a062] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:53:53,549 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-98) [eb8a062] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:53:56,558 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:53:58,558 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:53:58,558 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:53:58,558 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-3) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:53:58,559 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-3) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:53:58,559 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-3) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:53:58,567 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-3) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:54:01,575 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:54:02,560 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:54:02,560 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:54:02,560 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-57) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:54:02,561 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-57) [16b48795] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:54:02,562 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-57) [16b48795] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:54:02,569 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-57) [16b48795] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:54:05,576 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:54:07,576 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:54:07,576 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:54:07,576 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-70) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:54:07,577 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-70) [562b8b0f] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:54:07,578 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-70) [562b8b0f] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:54:07,584 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-70) [562b8b0f] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:54:10,592 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:54:11,578 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:54:11,578 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:54:11,578 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-19) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:54:11,579 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-19) [5a26db10] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:54:11,580 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-19) [5a26db10] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:54:11,587 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-19) [5a26db10] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:54:14,594 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:54:16,595 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:54:16,595 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:54:16,595 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-26) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:54:16,596 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-26) [13b06b4d] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:54:16,597 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-26) [13b06b4d] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:54:16,603 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-26) [13b06b4d] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:54:19,610 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:54:20,596 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:54:20,596 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:54:20,596 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-8) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:54:20,598 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-8) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:54:20,598 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-8) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:54:20,610 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-8) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:54:23,618 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:54:25,618 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:54:25,618 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:54:25,618 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-86) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:54:25,620 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-86) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:54:25,620 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-86) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:54:25,628 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-86) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:54:28,637 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:54:29,620 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:54:29,620 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:54:29,620 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-95) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:54:29,622 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-95) [4bd2250] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:54:29,622 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-95) [4bd2250] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:54:29,632 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-95) [4bd2250] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:54:32,640 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:54:34,640 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:54:34,640 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:54:34,640 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-89) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:54:34,641 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-89) [18caed52] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:54:34,642 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-89) [18caed52] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:54:34,651 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-89) [18caed52] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:54:37,658 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:54:38,642 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:54:38,642 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:54:38,642 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-87) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:54:38,643 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-87) [1dbf513e] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:54:38,644 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-87) [1dbf513e] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:54:38,651 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-87) [1dbf513e] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:54:41,658 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:54:43,658 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:54:43,658 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:54:43,658 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-3) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:54:43,659 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-3) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:54:43,660 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-3) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:54:43,668 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-3) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:54:46,674 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:54:47,660 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:54:47,660 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:54:47,660 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-33) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:54:47,662 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-33) [451c3558] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:54:47,663 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-33) [451c3558] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:54:47,672 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-33) [451c3558] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:54:50,683 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:54:52,683 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:54:52,683 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:54:52,683 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-53) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:54:52,684 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-53) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:54:52,684 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-53) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:54:52,692 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-53) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:54:55,699 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:54:56,685 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:54:56,685 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:54:56,685 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-62) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:54:56,686 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-62) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:54:56,686 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-62) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:54:56,694 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-62) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:54:59,701 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:55:01,701 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:55:01,701 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:55:01,701 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-20) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:55:01,702 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-20) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:55:01,703 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-20) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:55:01,711 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-20) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:55:04,717 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:55:05,703 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:55:05,703 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:55:05,703 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-92) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:55:05,705 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-92) [3c53e142] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:55:05,705 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-92) [3c53e142] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:55:05,717 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-92) [3c53e142] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:55:08,724 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:55:10,725 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:55:10,725 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:55:10,725 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-9) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:55:10,726 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-9) [5c6fc8e4] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:55:10,727 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-9) [5c6fc8e4] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:55:10,735 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-9) [5c6fc8e4] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:55:13,745 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:55:14,727 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:55:14,727 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:55:14,727 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-52) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:55:14,729 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-52) [7f575cd6] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:55:14,729 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-52) [7f575cd6] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:55:14,737 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-52) [7f575cd6] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:55:14,741 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-52) [7f575cd6] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:55:17,748 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:55:19,749 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:55:19,749 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:55:19,749 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-16) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:55:19,750 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-16) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:55:19,750 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-16) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:55:19,760 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-16) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:55:22,766 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:55:23,751 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:55:23,751 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:55:23,751 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-25) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:55:23,753 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-25) [5298f326] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:55:23,753 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-25) [5298f326] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:55:23,760 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-25) [5298f326] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:55:26,766 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:55:28,767 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:55:28,767 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:55:28,767 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-61) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:55:28,768 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-61) [20ef7a0c] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:55:28,768 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-61) [20ef7a0c] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:55:28,776 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-61) [20ef7a0c] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:55:31,782 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:55:32,768 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:55:32,768 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:55:32,768 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-40) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:55:32,769 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-40) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:55:32,769 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-40) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:55:32,776 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-40) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:55:35,785 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:55:37,785 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:55:37,785 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:55:37,785 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-14) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:55:37,786 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-14) [6343caea] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:55:37,786 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-14) [6343caea] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:55:37,794 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-14) [6343caea] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:55:40,801 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:55:41,787 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:55:41,787 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:55:41,787 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-26) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:55:41,789 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-26) [13b06b4d] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:55:41,789 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-26) [13b06b4d] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:55:41,796 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-26) [13b06b4d] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:55:44,803 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:55:46,804 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:55:46,804 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:55:46,804 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-38) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:55:46,805 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-38) [56c37b5] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:55:46,805 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-38) [56c37b5] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:55:46,812 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-38) [56c37b5] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:55:49,819 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:55:50,805 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:55:50,805 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:55:50,805 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-32) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:55:50,807 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-32) [56381f88] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:55:50,807 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-32) [56381f88] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:55:50,815 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-32) [56381f88] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:55:53,823 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:55:55,823 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:55:55,823 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:55:55,825 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-28) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:55:55,827 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-28) [60adf93f] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:55:55,827 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-28) [60adf93f] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:55:55,836 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-28) [60adf93f] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:55:58,847 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:55:59,825 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:55:59,825 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:55:59,825 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-89) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:55:59,827 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-89) [18caed52] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:55:59,827 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-89) [18caed52] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:55:59,839 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-89) [18caed52] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:56:02,847 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:56:04,847 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:56:04,847 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:56:04,847 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-75) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:56:04,849 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-75) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:56:04,849 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-75) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:56:04,856 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-75) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:56:07,863 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:56:08,849 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:56:08,849 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:56:08,849 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-57) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:56:08,850 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-57) [16b48795] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:56:08,851 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-57) [16b48795] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:56:08,858 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-57) [16b48795] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:56:11,865 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:56:13,865 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:56:13,865 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:56:13,865 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-61) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:56:13,868 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-61) [20ef7a0c] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:56:13,868 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-61) [20ef7a0c] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:56:13,876 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-61) [20ef7a0c] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:56:16,884 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:56:17,867 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:56:17,867 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:56:17,867 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-82) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:56:17,869 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-82) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:56:17,869 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-82) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:56:17,878 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-82) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:56:20,888 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:56:22,888 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:56:22,888 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:56:22,888 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-41) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:56:22,889 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-41) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:56:22,889 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-41) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:56:22,897 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-41) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:56:25,905 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:56:26,890 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:56:26,890 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:56:26,890 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-76) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:56:26,891 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-76) [1bdd13e4] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:56:26,892 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-76) [1bdd13e4] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:56:26,899 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-76) [1bdd13e4] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:56:29,907 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:56:31,907 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:56:31,907 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:56:31,909 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-38) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:56:31,910 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-38) [56c37b5] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:56:31,910 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-38) [56c37b5] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:56:31,917 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-38) [56c37b5] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:56:34,953 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:56:35,909 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:56:35,909 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:56:35,909 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-95) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:56:35,911 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-95) [4bd2250] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:56:35,911 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-95) [4bd2250] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:56:35,919 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-95) [4bd2250] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:56:38,926 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:56:40,927 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:56:40,927 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:56:40,927 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-24) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:56:40,928 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-24) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:56:40,928 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-24) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:56:40,937 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-24) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:56:43,946 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:56:44,929 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:56:44,929 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:56:44,930 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-79) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:56:44,931 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-79) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:56:44,931 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-79) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:56:44,938 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-79) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:56:47,946 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:56:49,946 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:56:49,946 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:56:49,946 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-58) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:56:49,947 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-58) [1a19a2ca] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:56:49,947 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-58) [1a19a2ca] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:56:49,955 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-58) [1a19a2ca] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:56:52,962 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:56:53,948 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:56:53,948 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:56:53,948 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-83) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:56:53,949 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-83) [6d722a1d] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:56:53,950 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-83) [6d722a1d] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:56:53,957 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-83) [6d722a1d] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:56:56,963 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:56:58,963 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:56:58,963 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:56:58,963 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-84) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:56:58,964 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-84) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:56:58,964 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-84) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:56:58,972 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-84) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:57:01,980 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:57:02,965 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:57:02,965 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:57:02,965 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-19) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:57:02,966 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-19) [5a26db10] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:57:02,967 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-19) [5a26db10] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:57:02,974 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-19) [5a26db10] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:57:05,982 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:57:07,982 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:57:07,982 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:57:07,982 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-26) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:57:07,983 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-26) [13b06b4d] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:57:07,983 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-26) [13b06b4d] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:57:07,991 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-26) [13b06b4d] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:57:10,998 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:57:11,984 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:57:11,984 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:57:11,984 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-13) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:57:11,986 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-13) [7246829] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:57:11,986 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-13) [7246829] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:57:11,995 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-13) [7246829] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:57:15,002 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:57:17,002 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:57:17,002 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:57:17,002 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-73) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:57:17,003 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-73) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:57:17,003 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-73) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:57:17,011 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-73) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:57:20,021 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:57:21,004 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:57:21,004 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:57:21,004 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-90) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:57:21,006 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-90) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:57:21,006 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-90) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:57:21,013 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-90) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:57:24,020 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:57:26,020 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:57:26,020 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:57:26,022 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-24) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:57:26,023 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-24) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:57:26,023 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-24) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:57:26,031 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-24) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:57:29,039 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:57:30,022 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:57:30,022 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:57:30,022 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-1) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:57:30,024 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-1) [30db9a86] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:57:30,024 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-1) [30db9a86] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:57:30,032 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-1) [30db9a86] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:57:33,039 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:57:35,040 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:57:35,040 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:57:35,040 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-3) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:57:35,041 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-3) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:57:35,041 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-3) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:57:35,049 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-3) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:57:38,056 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:57:39,042 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:57:39,042 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:57:39,042 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-29) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:57:39,044 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-29) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:57:39,044 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-29) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:57:39,052 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-29) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:57:42,060 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:57:44,060 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:57:44,060 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:57:44,060 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-84) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:57:44,061 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-84) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:57:44,061 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-84) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:57:44,069 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-84) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:57:47,076 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:57:48,062 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:57:48,062 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:57:48,062 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-41) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:57:48,064 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-41) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:57:48,064 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-41) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:57:48,075 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-41) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:57:51,086 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:57:53,086 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:57:53,086 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:57:53,086 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-94) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:57:53,087 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-94) [5dd0314d] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:57:53,087 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-94) [5dd0314d] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:57:53,096 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-94) [5dd0314d] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:57:56,103 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:57:57,088 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:57:57,088 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:57:57,088 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-50) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:57:57,090 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-50) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:57:57,090 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-50) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:57:57,098 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-50) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:58:00,106 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:58:02,106 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:58:02,106 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:58:02,106 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-88) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:58:02,107 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-88) [f89ebd8] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:58:02,108 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-88) [f89ebd8] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:58:02,115 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-88) [f89ebd8] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:58:05,123 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:58:06,108 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:58:06,108 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:58:06,108 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-52) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:58:06,110 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-52) [7f575cd6] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:58:06,110 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-52) [7f575cd6] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:58:06,118 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-52) [7f575cd6] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:58:09,125 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:58:11,125 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:58:11,125 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:58:11,125 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-16) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:58:11,126 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-16) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:58:11,126 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-16) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:58:11,134 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-16) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:58:14,143 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:58:15,127 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:58:15,127 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:58:15,127 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-25) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:58:15,128 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-25) [5298f326] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:58:15,129 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-25) [5298f326] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:58:15,139 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-25) [5298f326] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:58:15,143 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-25) [5298f326] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:58:18,150 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:58:20,150 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:58:20,150 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:58:20,150 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-54) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:58:20,151 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-54) [15e2b976] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:58:20,151 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-54) [15e2b976] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:58:20,158 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-54) [15e2b976] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:58:23,165 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:58:24,152 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:58:24,152 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:58:24,152 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-29) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:58:24,153 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-29) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:58:24,153 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-29) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:58:24,161 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-29) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:58:27,167 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:58:29,167 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:58:29,167 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:58:29,167 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-21) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:58:29,168 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-21) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:58:29,168 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-21) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:58:29,176 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-21) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:58:32,183 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:58:33,169 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:58:33,169 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:58:33,169 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-46) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:58:33,171 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-46) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:58:33,171 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-46) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:58:33,178 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-46) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:58:36,186 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:58:38,186 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:58:38,186 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:58:38,186 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-65) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:58:38,188 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-65) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:58:38,188 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-65) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:58:38,196 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-65) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:58:41,203 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:58:42,188 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:58:42,188 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:58:42,188 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-86) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:58:42,190 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-86) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:58:42,190 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-86) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:58:42,197 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-86) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:58:45,205 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:58:47,205 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:58:47,205 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:58:47,205 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-28) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:58:47,207 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-28) [60adf93f] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:58:47,207 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-28) [60adf93f] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:58:47,214 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-28) [60adf93f] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:58:50,220 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:58:51,207 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:58:51,207 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:58:51,207 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-89) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:58:51,209 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-89) [18caed52] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:58:51,209 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-89) [18caed52] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:58:51,216 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-89) [18caed52] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:58:54,223 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:58:56,223 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:58:56,223 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:58:56,223 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-79) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:58:56,224 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-79) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:58:56,224 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-79) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:58:56,233 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-79) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:58:59,242 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:59:00,225 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:59:00,225 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:59:00,225 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-96) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:59:00,228 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-96) [721bb546] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:59:00,228 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-96) [721bb546] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:59:00,238 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-96) [721bb546] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:59:03,246 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:59:05,246 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:59:05,246 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:59:05,246 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-33) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:59:05,249 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-33) [451c3558] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:59:05,249 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-33) [451c3558] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:59:05,259 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-33) [451c3558] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:59:08,268 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:59:09,248 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:59:09,248 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:59:09,248 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-82) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:59:09,251 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-82) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:59:09,251 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-82) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:59:09,261 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-82) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:59:12,269 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:59:14,269 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:59:14,269 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:59:14,270 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-21) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:59:14,272 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-21) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:59:14,272 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-21) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:59:14,280 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-21) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:59:17,288 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:59:18,272 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:59:18,272 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:59:18,272 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-51) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:59:18,274 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-51) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:59:18,274 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-51) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:59:18,281 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-51) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:59:21,289 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:59:23,289 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:59:23,289 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:59:23,289 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-38) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:59:23,290 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-38) [56c37b5] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:59:23,290 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-38) [56c37b5] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:59:23,298 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-38) [56c37b5] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:59:26,307 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:59:27,291 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:59:27,291 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:59:27,291 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-47) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:59:27,293 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-47) [4028fe5f] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:59:27,293 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-47) [4028fe5f] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:59:27,300 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-47) [4028fe5f] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:59:30,306 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:59:32,306 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:59:32,306 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:59:32,306 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-28) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:59:32,309 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-28) [60adf93f] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:59:32,309 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-28) [60adf93f] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:59:32,316 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-28) [60adf93f] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:59:35,324 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:59:36,308 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:59:36,308 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:59:36,308 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-16) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:59:36,311 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-16) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:59:36,311 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-16) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:59:36,322 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-16) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:59:39,330 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:59:41,330 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:59:41,330 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:59:41,330 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-58) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:59:41,331 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-58) [1a19a2ca] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:59:41,331 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-58) [1a19a2ca] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:59:41,339 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-58) [1a19a2ca] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:59:44,348 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:59:45,332 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:59:45,332 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:59:45,332 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-80) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:59:45,334 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-80) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:59:45,334 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-80) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:59:45,340 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-80) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:59:48,347 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:59:50,347 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:59:50,347 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:59:50,347 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-29) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:59:50,348 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-29) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:59:50,348 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-29) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:59:50,356 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-29) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:59:53,362 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:59:54,351 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:59:54,351 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:59:54,351 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-82) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:59:54,352 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-82) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 09:59:54,352 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-82) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:59:54,359 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-82) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:59:57,367 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 09:59:59,367 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:59:59,367 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:59:59,367 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-21) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 09:59:59,369 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-21) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 09:59:59,369 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-21) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 09:59:59,375 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-21) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 10:00:02,214 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-51) Attempting to update VMs/Templates Ovf. >2014-10-08 10:00:02,382 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 10:00:03,369 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 10:00:03,369 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 10:00:03,369 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-65) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 10:00:03,371 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-65) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 10:00:03,371 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-65) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 10:00:03,379 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-65) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 10:00:06,387 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 10:00:08,387 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 10:00:08,387 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 10:00:08,387 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-31) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 10:00:08,389 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-31) [2eda49c0] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 10:00:08,389 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-31) [2eda49c0] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 10:00:08,396 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-31) [2eda49c0] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 10:00:11,404 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 10:00:12,389 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 10:00:12,389 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 10:00:12,389 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-52) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 10:00:12,390 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-52) [7f575cd6] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 10:00:12,390 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-52) [7f575cd6] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 10:00:12,398 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-52) [7f575cd6] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 10:00:15,407 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 10:00:17,407 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 10:00:17,407 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 10:00:17,407 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-1) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 10:00:17,408 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-1) [30db9a86] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 10:00:17,409 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-1) [30db9a86] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 10:00:17,416 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-1) [30db9a86] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 10:00:20,424 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 10:00:21,409 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 10:00:21,409 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 10:00:21,409 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-25) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 10:00:21,411 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-25) [5298f326] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 10:00:21,411 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-25) [5298f326] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 10:00:21,420 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-25) [5298f326] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 10:00:24,427 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 10:00:26,427 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 10:00:26,427 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 10:00:26,427 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-10) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 10:00:26,428 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-10) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 10:00:26,428 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-10) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 10:00:26,435 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-10) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 10:00:29,474 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 10:00:30,430 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 10:00:30,430 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 10:00:30,430 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-100) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 10:00:30,432 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-100) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 10:00:30,432 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-100) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 10:00:30,440 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-100) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 10:00:33,446 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 10:00:35,446 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 10:00:35,446 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 10:00:35,446 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-82) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 10:00:35,447 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-82) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 10:00:35,447 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-82) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 10:00:35,455 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-82) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 10:00:38,463 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 10:00:39,448 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 10:00:39,448 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 10:00:39,448 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-34) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 10:00:39,451 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-34) [3e42dc0b] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 10:00:39,451 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-34) [3e42dc0b] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 10:00:39,461 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-34) [3e42dc0b] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 10:00:42,470 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 10:00:44,471 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 10:00:44,471 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 10:00:44,471 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-13) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 10:00:44,473 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-13) [7246829] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 10:00:44,473 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-13) [7246829] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 10:00:44,481 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-13) [7246829] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 10:00:47,488 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 10:00:48,473 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 10:00:48,473 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 10:00:48,473 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-73) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 10:00:48,475 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-73) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 10:00:48,475 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-73) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 10:00:48,482 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-73) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 10:00:51,492 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 10:00:53,492 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 10:00:53,492 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 10:00:53,492 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-74) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 10:00:53,493 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-74) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 10:00:53,494 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-74) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 10:00:53,502 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-74) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 10:00:56,510 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 10:00:57,494 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 10:00:57,494 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 10:00:57,494 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-24) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 10:00:57,496 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-24) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 10:00:57,496 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-24) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 10:00:57,506 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-24) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 10:01:00,515 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 10:01:02,515 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 10:01:02,515 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 10:01:02,515 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-79) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 10:01:02,517 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-79) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 10:01:02,517 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-79) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 10:01:02,526 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-79) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 10:01:05,535 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 10:01:06,517 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 10:01:06,517 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 10:01:06,517 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-54) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 10:01:06,519 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-54) [15e2b976] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 10:01:06,519 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-54) [15e2b976] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 10:01:06,530 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-54) [15e2b976] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 10:01:09,538 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 10:01:11,539 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 10:01:11,539 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 10:01:11,539 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-33) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 10:01:11,540 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-33) [451c3558] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 10:01:11,541 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-33) [451c3558] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 10:01:11,548 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-33) [451c3558] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 10:01:14,559 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 10:01:15,541 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 10:01:15,541 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 10:01:15,541 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-62) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 10:01:15,543 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-62) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 10:01:15,543 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-62) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 10:01:15,548 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-62) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 10:01:15,553 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-62) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 10:01:18,560 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 10:01:20,560 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 10:01:20,560 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 10:01:20,560 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-26) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 10:01:20,561 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-26) [13b06b4d] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 10:01:20,562 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-26) [13b06b4d] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 10:01:20,569 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-26) [13b06b4d] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 10:01:23,577 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 10:01:24,562 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 10:01:24,562 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 10:01:24,562 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-42) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 10:01:24,564 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-42) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 10:01:24,564 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-42) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 10:01:24,571 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-42) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 10:01:27,579 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 10:01:29,579 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 10:01:29,579 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 10:01:29,579 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-32) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 10:01:29,580 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-32) [56381f88] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 10:01:29,580 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-32) [56381f88] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 10:01:29,589 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-32) [56381f88] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 10:01:32,597 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 10:01:33,581 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 10:01:33,581 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 10:01:33,581 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-47) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 10:01:33,583 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-47) [4028fe5f] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 10:01:33,583 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-47) [4028fe5f] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 10:01:33,591 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-47) [4028fe5f] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 10:01:36,599 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 10:01:38,599 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 10:01:38,599 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 10:01:38,599 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-89) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 10:01:38,600 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-89) [18caed52] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 10:01:38,600 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-89) [18caed52] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 10:01:38,608 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-89) [18caed52] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 10:01:41,616 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 10:01:42,601 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 10:01:42,601 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 10:01:42,601 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-16) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 10:01:42,603 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-16) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 10:01:42,603 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-16) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 10:01:42,610 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-16) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 10:01:45,617 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 10:01:47,618 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 10:01:47,618 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 10:01:47,618 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-25) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 10:01:47,619 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-25) [5298f326] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 10:01:47,619 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-25) [5298f326] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 10:01:47,626 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-25) [5298f326] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 10:01:50,633 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 10:01:51,620 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 10:01:51,620 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 10:01:51,620 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-80) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 10:01:51,622 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-80) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 10:01:51,623 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-80) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 10:01:51,632 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-80) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 10:01:54,641 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 10:01:56,641 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 10:01:56,641 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 10:01:56,641 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-29) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 10:01:56,642 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-29) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 10:01:56,643 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-29) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 10:01:56,650 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-29) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 10:01:59,660 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 10:02:00,643 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 10:02:00,643 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 10:02:00,643 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-7) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 10:02:00,644 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-7) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 10:02:00,644 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-7) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 10:02:00,652 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-7) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 10:02:03,659 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 10:02:05,659 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 10:02:05,659 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 10:02:05,659 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-44) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 10:02:05,661 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-44) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 10:02:05,661 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-44) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 10:02:05,670 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-44) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 10:02:08,677 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 10:02:09,662 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 10:02:09,662 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 10:02:09,662 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-92) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 10:02:09,664 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-92) [3c53e142] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 10:02:09,664 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-92) [3c53e142] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 10:02:09,672 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-92) [3c53e142] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 10:02:12,679 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 10:02:14,679 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 10:02:14,679 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 10:02:14,680 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-32) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 10:02:14,682 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-32) [56381f88] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 10:02:14,682 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-32) [56381f88] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 10:02:14,688 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-32) [56381f88] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 10:02:17,695 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 10:02:18,681 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 10:02:18,681 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 10:02:18,681 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-28) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 10:02:18,683 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-28) [60adf93f] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 10:02:18,683 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-28) [60adf93f] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 10:02:18,690 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-28) [60adf93f] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 10:02:21,700 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 10:02:23,700 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 10:02:23,700 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 10:02:23,700 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-87) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 10:02:23,701 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-87) [1dbf513e] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 10:02:23,701 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-87) [1dbf513e] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 10:02:23,708 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-87) [1dbf513e] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 10:02:26,718 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 10:02:27,702 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 10:02:27,702 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 10:02:27,702 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-79) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 10:02:27,704 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-79) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 10:02:27,704 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-79) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 10:02:27,716 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-79) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 10:02:30,725 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 10:02:32,725 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 10:02:32,725 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 10:02:32,727 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-25) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 10:02:32,728 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-25) [5298f326] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 10:02:32,728 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-25) [5298f326] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 10:02:32,735 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-25) [5298f326] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 10:02:35,744 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 10:02:36,727 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 10:02:36,727 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 10:02:36,727 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-4) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 10:02:36,729 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-4) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 10:02:36,729 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-4) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 10:02:36,738 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-4) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 10:02:39,745 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 10:02:41,745 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 10:02:41,745 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 10:02:41,745 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-14) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 10:02:41,746 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-14) [6343caea] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 10:02:41,747 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-14) [6343caea] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 10:02:41,755 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-14) [6343caea] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 10:02:44,765 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 10:02:45,747 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 10:02:45,747 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 10:02:45,747 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-20) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 10:02:45,748 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-20) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 10:02:45,748 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-20) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 10:02:45,757 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-20) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 10:02:48,764 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 10:02:50,765 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 10:02:50,765 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 10:02:50,765 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-44) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 10:02:50,768 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-44) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 10:02:50,768 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-44) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 10:02:50,775 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-44) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 10:02:53,784 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 10:02:54,767 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 10:02:54,767 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 10:02:54,767 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-65) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 10:02:54,769 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-65) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 10:02:54,769 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-65) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 10:02:54,776 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-65) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 10:02:57,783 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 10:02:59,784 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 10:02:59,784 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 10:02:59,784 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-90) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 10:02:59,785 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-90) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 10:02:59,785 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-90) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 10:02:59,793 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-90) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 10:03:02,800 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 10:03:03,786 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 10:03:03,786 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 10:03:03,786 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-35) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 10:03:03,788 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-35) [1ab5e2c1] Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 10:03:03,788 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-35) [1ab5e2c1] Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 10:03:03,797 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-35) [1ab5e2c1] Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 10:03:06,806 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 10:03:08,806 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 10:03:08,806 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 10:03:08,806 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-16) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 10:03:08,808 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-16) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 10:03:08,808 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-16) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 10:03:08,816 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-16) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 10:03:11,824 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 10:03:12,808 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 10:03:12,808 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 10:03:12,808 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-39) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 10:03:12,810 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-39) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 10:03:12,810 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-39) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 10:03:12,816 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-39) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 10:03:15,823 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 10:03:17,823 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 10:03:17,823 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 10:03:17,823 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-40) Exception during connection: java.util.concurrent.ExecutionException: java.rmi.ConnectException: Connection timeout > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 10:03:17,824 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-40) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.rmi.ConnectException: Connection timeout >2014-10-08 10:03:17,824 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-40) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >2014-10-08 10:03:17,832 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-40) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.rmi.ConnectException: Connection timeout > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.rmi.ConnectException: Connection timeout > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:113) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 10:03:20,838 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to slot-7.rhev.lab.eng.brq.redhat.com/10.34.63.137 >2014-10-08 10:03:21,825 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 10:03:21,825 WARN [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 10:03:21,825 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler_Worker-84) Exception during connection: java.util.concurrent.ExecutionException: java.net.NoRouteToHostException: No route to host > at java.util.concurrent.FutureTask.report(FutureTask.java:122) [rt.jar:1.7.0_65] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:122) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:109) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:67) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:60) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:184) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:657) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:504) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) [:1.7.0_65] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: java.net.NoRouteToHostException: No route to host > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_65] > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:111) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$2.call(ReactorClient.java:97) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:26) [vdsm-jsonrpc-java-client.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_65] > at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:58) [vdsm-jsonrpc-java-client.jar:] > >2014-10-08 10:03:21,827 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-84) Command GetCapabilitiesVDSCommand(HostName = blade, HostId = 96620dcc-2509-48fe-841b-85247771c4fb, vds=Host[blade,96620dcc-2509-48fe-841b-85247771c4fb]) execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-10-08 10:03:21,827 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-84) Host blade is not responding. It will stay in Connecting state for a grace period of 120 seconds and after that an attempt to fence the host will be issued. >